Pith. sign in

REVIEW 3 major objections 5 minor 31 references

SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Popular multilingual benchmarks misjudge LLM performance on Swift, and a 28-problem hand-crafted benchmark shows small models drop most on Swift-specific features.

desk verdict SwiftEval is a useful Swift benchmark and defect catalog, but the central claim that score drops reflect language-specific features is confounded by unmatched problem sets. read the letter →

arxiv 2505.24324 v1 pith:5J4VGTBV submitted 2025-05-30 cs.LG cs.CLcs.PLcs.SE

Ivan Petrukha , Yana Kurliak , Nataliia Stulova This is my paper
classification cs.LGcs.CLcs.PLcs.SE
keywords SwiftcodegenerationbenchmarkLLMevaluationpass@1multilinguallanguage-specificfeaturesprogramsynthesiscontamination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that popular multilingual code benchmarks give a misleading picture of how well large language models write Swift. It documents specific translation errors in benchmarks such as HumanEval-XL and MultiPL-E, including wrong types, Python-style rounding assumptions, and tuple tasks that are unsolvable in Swift, and concludes that their Swift scores are not trustworthy. To address this, the paper introduces SwiftEval, a benchmark of 28 hand-crafted problems that exercise Swift-specific features such as optionals, protocols, generics, and closures. Evaluating 44 code-generation models, it finds clear score drops on these language-specific tasks, most sharply for smaller models, and a stronger model-size correlation than on HumanEval. If correct, this would mean that Python-centric benchmarks overstate Swift capability and that language-tailored benchmarks are needed for dependable multilingual evaluation.

What carries the argument

The load-bearing artifact is SwiftEval itself: 28 hand-written Swift problems, each with a natural language query, code context, an entrypoint for generation, and 3–5 unit tests, compiled with the official Swift compiler and executed on macOS. The evaluation protocol is pass@1 with 20 samples per problem and bootstrapped confidence intervals, and the analysis contrasts the model-size score trend on HumanEval (correlation 0.30) with that on SwiftEval (correlation 0.50). The human-authored problem set is what allows the paper to attribute score gaps to Swift-specific features rather than to translation artifacts, and the correlation contrast is what supports its reading that HumanEval scores partly reflect memorization.

What would settle it

Translate the same 28 SwiftEval problems into Python with unit tests implementing identical logic and evaluate the same 44 models; if the Python scores show the same drop pattern as Swift, the claim that Swift-specific features are the cause fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing multilingual benchmarks for Swift are insufficient or even irrelevant, so scores on them do not reflect actual Swift coding ability. Its evidence is SwiftEval, a 28-problem, hand-crafted Swift benchmark, plus evaluation of 44 models using pass@1 with 20 completions per problem and bootstrapped 95% confidence intervals. On SwiftEval, most models show substantial score drops relative to HumanEval, with the largest drops concentrated in smaller models, and the correlation between model size and score rises from 0.30 on HumanEval to 0.50 on SwiftEval. The paper also reports that 86% of failed evaluations were compilation errors rather than unit-test failures, indicating that models struggle with Swift's syntax and type system, not just with problem logic.

Load-bearing premise

The paper assumes the score gap between HumanEval and SwiftEval comes from Swift-specific features, even though the two problem sets are not matched in difficulty and no Python version of the same SwiftEval problems is tested.

Editorial extensions

If this is right

  • A leaderboard built on translated Python-to-Swift prompts will likely mis-rank models' real Swift ability; SwiftEval orders models differently from HumanEval.
  • Small code models are especially unreliable on Swift-specific constructs, so strong Python pass@1 scores should not be used to infer Swift proficiency.
  • The high compilation-failure rate means syntax and type-system errors dominate over logic errors in Swift generation, pointing training-data efforts toward Swift source coverage.
  • Future multilingual benchmark builders should verify translated prompts against each target language's type system and runtime semantics before reporting scores.
  • Small, hand-crafted micro-benchmarks are a viable alternative to large translated benchmarks for languages underrepresented in training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test would be a Python translation of the same 28 SwiftEval prompts; if the Python versions show the same score drop pattern, part of the gap is problem difficulty rather than Swift-specificity, a confound the paper's design cannot fully rule out.
  • The same quality-over-quantity recipe could be applied to other statically typed or compiled languages, producing small idiomatic benchmarks for languages like Rust, Go, or C++.
  • The compilation-error rate suggests a cheap diagnostic for compiled languages: requiring code to compile before scoring may be a strong signal of model familiarity with target-language syntax.
  • The higher size-score correlation on SwiftEval could serve as a practical contamination probe: benchmarks whose scores track model scale are less likely to be memorized from pretraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that existing multilingual code-generation benchmarks are inadequate for Swift and presents SwiftEval, a new benchmark of 28 hand-crafted Swift problems. It reports pass@1 results for 44 code LLMs, compares these with the same models' HumanEval (Python) scores, and observes that scores drop more sharply on SwiftEval for smaller models. The paper also reports a higher correlation between model size and SwiftEval scores than between model size and HumanEval scores, interpreting this as evidence that HumanEval suffers from memorization and that SwiftEval better reflects genuine language-specific difficulty.

Significance. The paper makes a useful contribution by drawing attention to real defects in machine-translated multilingual benchmarks and by releasing a manually curated Swift benchmark with publicly available problem data and results. The concrete examples of translation errors in HumanEval-XL and MultiPL-E, such as the Python-specific rounding behavior and fixed-size tuple issues, are informative and actionable for benchmark builders. The evaluation of 44 models is a substantial empirical effort, and the publication of the artifact enables reproducibility. However, the central claim that the observed score drops are caused by Swift-specific language features is not established by the reported experiments, because the comparison is between two different problem sets that vary in language, task type, difficulty, and domain. The benchmark itself is a plausible contribution even if the comparative interpretation is revised.

major comments (3)
  1. [Abstract and Section IV, Table I] The headline claim that 'LLM scores drop for problems requiring language-specific features' is not supported by the presented evidence. Table I compares HumanEval (164 function-level algorithmic Python problems) with SwiftEval (28 class-level and function-level practical Swift problems), so language is confounded with problem content, difficulty, task type, and domain. A model could score lower on SwiftEval simply because the problems are harder or less familiar, independently of Swift-specific features. To support the language-specificity interpretation, the authors need a matched control: for example, a carefully translated HumanEval-to-Swift set with manual validation, or a set of SwiftEval problems also posed in Python with equivalent difficulty. Without such a control, the central claim should be weakened to 'scores drop on this new benchmark' rather than attributed to language features.
  2. [Section IV, Figures 1 and 2] The argument that a weak correlation (0.30) between model size and HumanEval score 'supports the point of potential benchmark memorization' is not valid. Correlation with model size is not a measure of memorization; a near-flat trend could arise from easy-problem saturation, prompt-format differences, contamination, or non-linear scaling behavior, none of which are controlled for here. The paper does not perform any contamination checks, such as n-gram overlap analysis or testing on rephrased problems, so the memorization claim is speculative. The higher SwiftEval correlation (0.50) is also a weak basis for confidence, since it is computed over a small number of models with heterogeneous architectures, training data, and prompt templates.
  3. [Section V and Section VI] The observation that 86% of SwiftEval failures are compilation errors (12,518 of 14,671) is important but not analyzed. A high compilation-failure rate could indicate that the generated code is syntactically or semantically incompatible with Swift, which would support a language-difficulty interpretation; however, it could also indicate problems with the provided code entrypoint, incomplete or ambiguous prompts, or strict compiler settings. The paper should include an error analysis that categorizes compilation failures (e.g., missing imports, type mismatches, incorrect API usage, entrypoint mismatches) and reports representative examples. Without this analysis, the interpretation that the benchmark isolates language-specific feature difficulty is incomplete.
minor comments (5)
  1. [Section IV, Evaluation setup] The description 'token probability 0.95' should be clarified as top-p or nucleus sampling; the exact decoding configuration matters for reproducibility.
  2. [Section IV, Evaluation setup] The calculation of pass@1 with 20 completions should be described more precisely; it would be helpful to state whether the reported value is the unbiased estimator from Chen et al., the mean over problems of the empirical pass rate, or another quantity.
  3. [Section II, MultiPL-E discussion] The phrase 'operator reserved keyword usage as function argument label' is too terse to be understood without an example; adding a short code snippet would substantially improve this critique.
  4. [Table I] The color-coding described in the caption is not visible in the text version; the table should include a legend or otherwise make the performance-change categories accessible to readers in monochrome.
  5. [Section III and Section VI] The paper states that SwiftEval 'currently contains 28 problems' and that results 'may vary slightly when more problems are added'; given the small sample size, the uncertainty is likely larger than 'slightly', and the authors should report standard errors or confidence intervals that reflect the number of problems, not just the number of samples per problem.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SwiftEval is an external benchmark artifact and the reported pass@1 scores are measurements, not fitted inputs or self-citation-derived quantities.

full rationale

The paper's derivation chain is not circular. SwiftEval is defined externally as a fixed set of 28 hand-crafted Swift problems with unit tests, and the pass@1 scores are measurements on that artifact. There are no fitted parameters, no equation-level reductions, and no load-bearing self-citations; the cited works are third-party benchmarks and models (HumanEval, MultiPL-E, DeepSeek-Coder-V2, Qwen2.5-Coder, etc.). The central claim that scores drop more on Swift-specific features is an interpretive comparison between HumanEval (Python) and SwiftEval (Swift) results. That comparison has a validity threat: the two problem sets are not matched for difficulty, so the observed gap could reflect problem difficulty or domain differences rather than language-specific features. However, this is a correctness/validity concern, not circularity, because the SwiftEval scores are not constructed from the conclusion; they are independent measurements on a pre-defined benchmark. The paper's own limitation section acknowledges the small 28-problem scale, further indicating that the claim is an empirical observation rather than a definitional tautology. No quoted passage exhibits a reduction of the conclusion to its inputs, so no circular step can be identified. Score 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest on five implicit assumptions: benchmark validity, comparability of HumanEval, accuracy of the critique, the memorization inference, and sample-size sufficiency. No free parameters are fitted; the benchmark is a fixed artifact.

free parameters (1)
  • Evaluation hyperparameters (temperature, top_p, number of samples) = T=0.2, p=0.95, n=20
    Chosen by hand following prior pass@1 estimation practice (Section IV); they affect variance estimates but not the benchmark content.
assumptions (5)
  • domain assumption The 28 SwiftEval problems are correct, unambiguous, and representative of Swift-specific programming.
    Problems were written by one industry engineer and validated only by unit tests within the benchmark (Section III); no external human validation is reported.
  • domain assumption HumanEval scores are a valid baseline for comparing language-specific difficulty.
    The paper compares HumanEval (Python) to SwiftEval (Swift) without controlling for problem difficulty (Section IV, Table I).
  • domain assumption Documented defects in HumanEval-XL and MultiPL-E accurately characterize those benchmarks.
    The critique (Section II) is based on examples such as AnyHashable usage and fixed-size tuple issues; no systematic error-rate analysis is provided.
  • domain assumption A weak size-score correlation on HumanEval indicates memorization or contamination.
    Section IV infers potential memorization from r=0.30, but this inference is not tested against contamination data.
  • standard math Pass@1 estimates from 20 samples per problem are stable.
    The paper relies on prior work [14] showing pass@1 stabilizes around 20 samples (Section IV).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation." pith.science (2026). https://pith.science/paper/5J4VGTBV

@misc{pith2026250524324,
  author       = {Pith},
  title        = {Pith review of: SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5J4VGTBV}},
  note         = {Machine review of arXiv:2505.24324}
}
read the original abstract

In recent years, large language models (LLMs) have showcased significant advancements in code generation. However, most evaluation benchmarks are primarily oriented towards Python, making it difficult to evaluate other programming languages, such as Swift, with high quality. By examining widely established multilingual benchmarks like HumanEval-XL and MultiPL-E, we identified critical issues specific to their Swift components, making them insufficient or even irrelevant for assessing LLM coding capabilities on Swift. Unlike these existing approaches, which prioritize rapid scaling and generalization by automatically translating Python-centric benchmarks with LLMs, we adopt a quality-over-quantity methodology. We present SwiftEval, the first Swift-oriented benchmark consisting of 28 carefully hand-crafted problems, and evaluate 44 popular Code LLMs on it. Our results show significant LLM scores drop for problems requiring language-specific features, most noticeable in the models of smaller sizes.

Figures

Figures reproduced from arXiv: 2505.24324 by the authors.

Figure 1
Figure 1. Performance on HumanEval (Python) Across Model Sizes [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Performance on SwiftEval (Swift) Across Model Sizes [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 14 canonical work pages

  1. [1]

    Evaluating Large Language Models Trained on Code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-V...

    arXiv 2021

  2. [2]

    CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,” Sept. 2022

    work page 2022

  3. [3]

    A System- atic Evaluation of Large Language Models of Code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A System- atic Evaluation of Large Language Models of Code,” May 2022. arXiv:2202.13169

    arXiv 2022

  4. [4]

    DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence,” Jan. 2024. arXiv:2401.14196

    arXiv 2024

  5. [5]

    Code Llama: Open Foundation Models for Code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, “Code Llama: Open Foundation Models for Code,” Jan

    work page

  6. [6]

    Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs,

    Y . Wang and H. Li, “Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14015–14023, May 2021

    work page 2021

  7. [7]

    Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code,

    R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, “Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , (Lisbon Portugal), pp. 1–13, ACM, Apr. 2024

    work page 2024

  8. [8]

    Few-shot training LLMs for project-specific code-summarization

    T. Ahmed and P. Devanbu, “Few-shot training LLMs for project-specific code-summarization,” Sept. 2022. arXiv:2207.04237

    work page Pith review arXiv 2022

Show all 31 references
  1. [9]

    Program Synthesis with Large Language Models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program Synthesis with Large Language Models,” Aug. 2021. arXiv:2108.07732 [cs]

    2021 arXiv

  2. [10]

    Multi-lingual Evaluation of Code Generation Models,

    B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y . Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V . Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B....

  3. [11]

    Large Language Models Meet NL2Code: A Survey,

    D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, Y . Wang, and J.-G. Lou, “Large Language Models Meet NL2Code: A Survey,” May 2023. arXiv:2212.09420

    2023 arXiv

  4. [12]

    Measuring Coding Challenge Competence With APPS,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring Coding Challenge Competence With APPS,” Nov. 2021. arXiv:2105.09938

    2021 arXiv

  5. [13]

    CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,” Feb. 2023. arXiv:2203.13474

    2023 arXiv

  6. [14]

    MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda, “MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation,” Dec

  7. [15]

    HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generaliza- tion,

    Q. Peng, Y . Chai, and X. Li, “HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generaliza- tion,” Mar. 2024. arXiv:2402.16694 [cs]

    2024 arXiv

  8. [17]

    The RedMonk Programming Language Rankings: June 2024,

    S. O’Grady, “The RedMonk Programming Language Rankings: June 2024,” Sept. 2024

    2024

  9. [18]

    BLEU: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, (Philadelphia, Pennsylvania), p. 311, Association for Computation...

    2001

  10. [19]

    CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,” Sept. 2020. arXiv:2009.10297 [cs]

    2020 arXiv

  11. [20]

    Rethinking Benchmark and Contamination for Language Models with Rephrased Samples,

    S. Yang, W.-L. Chiang, L. Zheng, J. E. Gonzalez, and I. Stoica, “Rethinking Benchmark and Contamination for Language Models with Rephrased Samples,” Nov. 2023. arXiv:2311.04850 [cs]

    2023 arXiv

  12. [21]

    DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W.- t. Yih, D. Fried, S. Wang, and T. Yu, “DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation,” Nov. 2022. arXiv:2211.11501

    2022 arXiv

  13. [22]

    ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation,” Aug. 2023. arXiv:2308.01861

    2023 arXiv

  14. [23]

    Cross- CodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion,

    Y . Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang, “Cross- CodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion,” Nov. 2023. arXiv:2310.11248

    2023 arXiv

  15. [24]

    JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models,

    J. Cao, Z. Chen, J. Wu, S.-c. Cheung, and C. Xu, “JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models,” Oct. 2024. arXiv:2406.12902

    2024 arXiv

  16. [25]

    mHumanEval – A Multilingual Benchmark to Evaluate Large Language Models for Code Generation,

    N. Raihan, A. Anastasopoulos, and M. Zampieri, “mHumanEval – A Multilingual Benchmark to Evaluate Large Language Models for Code Generation,” Oct. 2024. arXiv:2410.15037

    2024 arXiv

  17. [26]

    DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence,

    DeepSeek-AI, Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Y . Li, H. Gao, S. Ma, W. Zeng, X. Bi, Z. Gu, H. Xu, D. Dai, K. Dong, L. Zhang, Y . Piao, Z. Gou, Z. Xie, Z. Hao, B. Wang, J. Song, D. Chen, X. Xie, K. Guan, Y . You, A. Liu, Q. Du, W. Gao, X. Lu, Q. Chen, ...

    2024 arXiv

  18. [27]

    Qwen2.5-Coder Technical Report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y . Fan, Y . Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y . Miao, S. Quan, Y . Feng, X. Ren, X. Ren, J. Zhou, and J. Lin, “Qwen2.5-Coder Technical Report,” Nov. 2024. arXiv:2409.12186

    2024 arXiv

  19. [28]

    Codestral: Hello, World!,

    M. AI, “Codestral: Hello, World!,” May 2024

    2024

  20. [29]

    CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?,

    V . Bhargava, R. Ghosh, and D. Dutta, “CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?,” Dec. 2024. arXiv:2412.02735

    2024 arXiv

  21. [30]

    HumanEval.jl,

    Yi Team, “HumanEval.jl,” Feb. 2024. original-date: 2024-02- 07T11:42:55Z

    2024

  22. [31]

    Rust Compiling Benchmark,

    Gusanidas, “Rust Compiling Benchmark,” Nov. 2024. original-date: 2024-11-21T23:10:10Z. APPENDIX ACKNOWLEDGEMENTS We thank the Armed Forces of Ukraine for providing secu- rity to complete this work. We thank the anonymous reviewers for providing valuable feedback and suggestion...

    2024

  23. [2022]

    arXiv:2208.08227 [cs]

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.