Pith. sign in

REVIEW 5 major objections 5 minor 52 references

CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark finds even the best LLMs fail roughly one in five cross-language IPC coding tasks.

desk verdict CrossPL builds a genuinely new benchmark for cross-language IPC code generation, but its credibility rests on an unvalidated FSM checker in place of execution. read the letter →

arxiv 2507.19904 v1 pith:IOT3FGKL submitted 2025-07-26 cs.SE cs.AI

Zhanhang Xiong , Dongxia Wang , Yuekang Li , Xinyuan An , Wenhai Wang This is my paper
classification cs.SEcs.AI
keywords cross-programminglanguagecodegenerationbenchmarkinter-processcommunicationfinitestatemachinevalidationlargemodelsmultilingualsoftwaresystemspass@kevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CrossPL, a benchmark of 1,982 tasks that ask an LLM to generate working inter-process communication (IPC) code connecting components written in different programming languages. Its purpose is to test a capability existing benchmarks ignore: not translating code, but producing code in one language that must interoperate with code in another. Across 20 LLMs and seven IPC techniques, the authors find that even the strongest model passes only 79.74% of tasks at pass@1 and most models stay below 70%, so correct cross-language glue code is not yet reliable. If the benchmark is sound, it gives the field a measurable target for a real-world skill that matters for multi-language software systems.

What carries the argument

The load-bearing object is the finite state machine (FSM) encoding of an IPC interaction pattern, such as import networking library, create server socket, bind port, listen, accept, exchange data, and close. The 156 FSMs, extended from a coarser eight-FSM toolkit, describe expected API call sequences per language and technique, skip code comments, and carry semantic descriptions of each step. These same FSMs do three jobs: they locate candidate CPL snippets in repositories, validate the extracted snippets during benchmark construction, and score LLM-generated code during evaluation. Because they encode protocol-required call order, they can catch missing steps such as a gRPC server that never binds a port.

What would settle it

Take a random sample of CrossPL tasks whose generated code passed FSM validation, compile or run each snippet together with its real counterpart process, send actual messages, and compare the observed success rate with the FSM pass rate; a large drop would show the validator overestimates model capability.

Watch

Extended reading notes

Core claim

The paper claims to present the first benchmark dedicated to LLM generation of cross-programming-language (CPL) interoperating code. CrossPL contains 1,982 IPC tasks mined from 19,169 multi-language GitHub repositories, covering Java, Python, Go, JavaScript, PHP, and C++ across seven IPC techniques. Correctness is judged by 156 hand-crafted finite state machines that encode the expected API call sequence for each IPC scenario. The central empirical result is that no evaluated model reliably generates such code: the best pass@1 is 79.74% (GLM4-plus), most models fall below 70%, performance varies sharply by language and protocol, and reasoning modes that simulate thinking do not help and sometimes hurt.

Load-bearing premise

The FSM check assumes that generating the expected API calls in the right order is sufficient for the code to work, and it never compiles or runs the generated code.

Editorial extensions

If this is right

  • A reliable capability gap is now measurable: no evaluated model reaches 80% pass@1 on CPL IPC tasks, and most models sit below 70%.
  • CrossPL gives the community a fixed 1,982-task testbed for comparing models on cross-language interoperability, broken down by language and by IPC technique.
  • The per-technique results isolate where models fail most: gRPC is comparatively easy, while Pipe and HTTP tasks drag scores down.
  • The Qwen3 family results indicate that neither larger scale nor thinking mode is a reliable lever for CPL IPC code generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the FSM validator is more permissive than real execution, the true success rates are likely lower than the reported pass@1 numbers, making the gap larger rather than smaller.
  • The FSM descriptions could be reused as structured generation constraints or as a reward signal in RL fine-tuning, a direction the paper does not explore.
  • Because FFI (foreign function interface) is deliberately excluded, the benchmark covers only one half of cross-language interoperability; models might behave differently on FFI tasks that require type mapping and runtime linking.
  • The thinking-mode result suggests current reasoning modes are tuned for general problem-solving rather than protocol compliance; a direct test would be to compare error types, such as missing steps versus wrong syntax, between base and thinking modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. CrossPL is a benchmark of 1,982 tasks intended to test LLMs on generating cross-programming-language (CPL) interoperating code. The tasks are derived from 19,169 multi-language GitHub repositories: 156 hand-crafted FSMs locate IPC interaction points, and an LLM-based pipeline (DeepSeek-V3) extracts snippets, generates instructions, and validates snippets. The authors evaluate 20 LLMs using FSM-based validation and report pass@1/pass@5 across six languages and seven IPC techniques. They find that even the best model (GLM4-plus) reaches only 79.74% pass@1, with most models below 70%, and conclude that current LLMs struggle with CPL/IPC generation, that performance varies by language and technique, and that thinking mode does not reliably help.

Significance. If the validity threats were resolved, CrossPL would be a useful addition: it is, to my knowledge, the first large-scale benchmark targeting IPC code generation in multi-language project contexts, and it compares 20 models on a substantial task set with public data and code. The paper is also honest about some limitations (uneven samples, no FFI, RQ3 scope). However, the benchmark's central numbers depend on two premises that are not established: that the FSM matcher measures functional correctness, and that the tasks are genuinely cross-language rather than single-language IPC exercises. These are not minor caveats; they affect the interpretation of every table and the main contribution claim. The strengths are the scale, the multi-language/multi-technique coverage, and the transparent pipeline.

major comments (5)
  1. [Section III-C (Algorithm 1, line 24)] The paper repeatedly states that FSM-based validation 'validates functional correctness' (Abstract and Section III-C), but the FSMs are static API-call sequence matchers, not execution. No experiment in the manuscript compares FSM verdicts with compilation, runtime behavior, or human judgment. Code that follows the enumerated steps can still have wrong types, wrong ports, or a broken peer setup and pass; correct code that uses an equivalent but non-enumerated API sequence can fail. Because the same family of FSMs is used both to accept extracted snippets during benchmark construction (Algorithm 2, A3) and to score generated code, any FSM blind spot biases both the dataset and the evaluation in the same direction. The pass@1/pass@5 numbers in Tables I-IV are therefore best interpreted as protocol-template match rates, not functional correctness rates, and the claim should be either renamed or backed by a stratified validation study (e.g., executing a sample of accepted and rejected outputs, or human annotation) with agreement statistics.
  2. [Section III-B and Fig. 7] The tasks in CrossPL appear to be single-language IPC implementations. The metadata recorded for each instance is (p_i, τ_i, θ_i, L_i, σ_i, K_i) with a single language L_i, and the example in Fig. 7 asks for a Python gRPC server setup only. The 'cross-language' property is inferred from the multi-language repository in which the snippet was found, not from the task itself, and the model is not required to know or generate any counterpart in another language. This raises a construct-validity issue for the title claim: the benchmark measures the ability to generate boilerplate IPC code in one language, which is related to but distinct from generating code that interoperates with a specific other-language component. The authors should either include cross-language context in the task (e.g., the other side's API or protocol contract) or reframe the contribution as an IPC-code benchmark for multi-language software contexts.
  3. [Section V-A] Pass@5 is computed with a minimum sample size of N = 5. For k = 5 and n = 5, the unbiased pass@k estimator reduces to 1 whenever at least one of the five samples passes (since C(5-c,5) = 0 for c > 0), so the reported pass@5 is an 'any-of-five' success rate, not the standard pass@5 used in HumanEval and later work. This inflates all pass@5 values and makes comparisons with other benchmarks meaningless. The authors should use n > k (e.g., n = 20) and report the estimator, or explicitly rename the metric and discuss its properties.
  4. [Section V-B, Tables I-IV] The cross-language and cross-technique comparisons are descriptive only: no confidence intervals, standard errors, or significance tests are reported, despite highly unbalanced subset sizes (e.g., C++ has 51 tasks, UDP has 92, Java has 615). The claimed patterns 'C++ best, Go worst' and 'gRPC best, Pipe worst' could therefore be driven by task difficulty, FSM strictness, or noise rather than model capability. Because the FSM acceptor is hand-built per technique and language, acceptance thresholds are unlikely to be calibrated across cells; the paper should report per-cell sample sizes with confidence intervals and test the key comparisons.
  5. [Section V-D, Tables V-VI] The RQ3 conclusion that 'thinking mode often reduces performance' is not supported by the reported numbers. In Table V, thinking mode improves Qwen3-8b (52.76 to 55.64) and Qwen3-30b-a3b (61.28 to 63.39), while it hurts Qwen3-4b (59.14 to 54.44) and Qwen3-235b (75.02 to 72.92); the remaining results are mixed. No significance tests or effect sizes are given. Similarly, the 'deviation from scaling law' claim is based on descriptive means without addressing variance or task imbalance. These claims should be rephrased as observations or supported with statistical analysis.
minor comments (5)
  1. [Section V-B and Table IV] The text states that pass@5 'consistently exceed their pass@1 counterparts across all task subcategories,' but Table IV shows Gemma-7b with pass@5 below pass@1 on HTTP (43.31% vs 47.50%) and Message Queue (46.68% vs 50.00%). This contradiction should be fixed.
  2. [Tables I-II] There are typos in model names and sizes: 'Lamma3-8b-instruct' appears in Tables I-IV, 'Qwen2.5-coder-32b-insrtuct' appears in Fig. 5, and the size for Lamma3-8b in Table II is listed as 70B while Table I says 8B. Please correct these.
  3. [Section III-B] The manuscript refers to supplementary details via 'in here' placeholders (e.g., prompt templates and additional benchmark details). These links are not resolved in the submitted text, which makes the pipeline difficult to reproduce. Please provide a complete appendix or stable URLs.
  4. [Section VI] The Limitations section lists uneven samples, missing FFI, and Qwen3-only RQ3, but it does not mention the most consequential threats discussed above: the lack of execution-based validation for the FSM checker and the single-language nature of the tasks. The section should be updated to acknowledge these.
  5. [Section I] The claim that CrossPL is 'the first benchmark' for CPL code generation would be more convincing with a systematic comparison of existing benchmarks that include IPC or multi-language interaction tasks (e.g., web/API generation benchmarks), rather than only multilingual translation benchmarks.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FSM oracle is fixed, externally grounded, and not fitted to model outputs.

full rationale

The paper's reported pass@k numbers are produced by FSM-based validation (Algorithm 1, lines 22-28), using 156 FSMs hand-crafted from official protocol documentation (Section III-A, III-C). The same FSMs are used to filter canonical snippets during benchmark construction (Algorithm 2, line 14) and to grade generated code during evaluation, but this is an intentional, fixed evaluation oracle rather than a circular step: the FSMs are defined before any evaluated model is run, are not tuned on model outputs, and are not derived from the benchmark's pass rates. No parameter is fitted to a subset of data and then 'predicted' on a closely related quantity; the FSM matching criterion is static across all models and tasks. The paper's citation of PolyFax [38] for the FSM idea is an external prior-work citation, not a self-citation, and the paper's contribution (156 finer-grained FSMs, the LLM extraction pipeline, and the 1,982-task benchmark) is additional content rather than a reduction to that citation. The limitation that FSM matching may not fully capture runtime functional correctness (Section VI) is a construct-validity threat, not a circularity: it does not make any reported result equal to its inputs by construction. The honest limitation statements about uneven samples, missing FFI, and Qwen3-only RQ3 further support that the empirical claims are presented as measurements against a stated oracle, not as a derivation from the oracle's definition.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The benchmark construction uses hand-chosen repository filters, an LLM temperature schedule, and 156 hand-crafted FSMs. The findings rely on domain assumptions about the validity of FSM-based scoring, the quality of LLM-generated instructions, and the representativeness of the selected open-source projects. No new physical or theoretical entities are introduced.

free parameters (5)
  • Repository star count bounds = 1000 to 30000
    Algorithm 1 uses these bounds to filter GitHub projects; they determine the 19,169-repo pool and therefore the composition of mined tasks.
  • Number of languages per project bounds = 2 to 5
    Algorithm 1 uses these bounds to define multi-language projects; they shape which repositories are considered MPL and thus which CPL snippets are mined.
  • LLM extraction temperature schedule = 0.1 * k for k=0..5
    Algorithm 2 retries snippet extraction up to five times with temperatures 0.1 to 0.5; this affects which snippets become benchmark tasks.
  • Pass@5 sampling temperature and top-p = temperature=0.2, top-p=0.95, N=5
    Section V-A fixes these for pass@5 evaluation; different sampling settings would change the numbers.
  • Number of hand-crafted FSMs = 156
    The authors designed 156 FSMs to detect and validate IPC patterns; this granularity defines what counts as correct CPL code.
assumptions (6)
  • domain assumption FSM-based matching is a valid proxy for functional correctness
    Section III-C states "Correctness is assessed by matching the generated code against the predefined FSMs to ensure protocol compliance." This equivalence is asserted, not validated against execution or human judgment.
  • domain assumption LLM-generated task instructions are clear and unambiguous
    Section III-B: instructions are generated by DeepSeek-V3 and only randomly sampled and manually inspected. Poor instructions would mis-measure model ability.
  • domain assumption The selected GitHub repositories are representative of real-world CPL usage
    Section III-A filters by star count (1000-30000) and language count (2-5); this selection shapes the benchmark and may not generalize to all multi-language systems.
  • domain assumption IPC is the primary CPL integration mechanism; FFI can be excluded
    Section I: "CrossPL focuses primarily on IPC-based interactions." FFI is deferred to future work, so type mapping, linking, and platform-specific setup are not covered.
  • domain assumption The extracted snippets are correct examples of CPL code
    Section III-B: snippets are selected if they match FSMs, but the original repository code is not independently verified for correctness, and trimming to a minimal snippet may omit context needed for actual interoperability.
  • domain assumption The seven IPC techniques are representative of CPL interoperability
    Section III-C and IV: the techniques (HTTP, TCP, UDP, WebSocket, Pipe, gRPC, Message Queue) are chosen from documentation and repo mining; other important mechanisms are not included.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation." pith.science (2026). https://pith.science/paper/IOT3FGKL

@misc{pith2026250719904,
  author       = {Pith},
  title        = {Pith review of: CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOT3FGKL}},
  note         = {Machine review of arXiv:2507.19904}
}
read the original abstract

As large language models (LLMs) become increasingly embedded in software engineering workflows, a critical capability remains underexplored: generating correct code that enables cross-programming-language (CPL) interoperability. This skill is essential for building complex systems that integrate components written in multiple languages via mechanisms like inter-process communication (IPC). To bridge this gap, we present CrossPL, the first benchmark designed to systematically evaluate LLMs' ability to generate CPL-interoperating code. CrossPL comprises 1,982 tasks centered around IPC, covering six widely-used programming languages and seven representative CPL techniques. We construct this benchmark by (i) analyzing 19,169 multi-language GitHub repositories using 156 hand-crafted finite state machines (FSMs), and (ii) developing an LLM-based pipeline that automatically extracts CPL code snippets, generates task instructions, and validates functional correctness. We evaluate 14 state-of-the-art general-purpose LLMs and 6 code-oriented LLMs released in the past three years on CrossPL via FSM-based validation. Results reveal that even the best-performing models struggle with CPL scenarios, underscoring the need for more targeted research in this space. Our benchmark and code are available at: https://anonymous.4open.science/r/crosspl-2814.

Figures

Figures reproduced from arXiv: 2507.19904 by the authors.

Figure 1
Figure 1. An example of IPC (Socket) between Python and C++. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework for CPL Interoperating Code Analysis, Extraction, Generation and Evaluation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of FSM-modeled CPL interoperating. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: a summarizes the distribution in CrossPL from differ￾ent perspectives. Overall it covers six programming languages and seven IPC technologies, comprising a total of 1982 tasks. Among the programming languages, Java accounts for the highest proportion of IPC-related tas…
Figure 5
Figure 5. Figure 5: Radar Plot of Pass@1 across different programming [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Boxplot of pass@1 performance of Qwen3 series models. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Case Analysis: the 495-th instance in the benchmark. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 28 canonical work pages

  1. [1]

    How are multilingual systems constructed: Characterizing language use and selection in open-source multilingual software,

    W. Li, A. Marino, H. Yang, N. Meng, L. Li, and H. Cai, “How are multilingual systems constructed: Characterizing language use and selection in open-source multilingual software,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 3, pp. 1–46, 2024

    work page 2024

  2. [2]

    Programming language trends in open source development: An evaluation using data from all production phase sourceforge projects,

    D. P. Delorey, C. D. Knutson, and C. Giraud-Carrier, “Programming language trends in open source development: An evaluation using data from all production phase sourceforge projects,” in Second International Workshop on Public Data about Software Development (WoPDaSD’07), 2007, pp. 1–5

    work page 2007

  3. [3]

    A large scale study of programming languages and code quality in github,

    B. Ray, D. Posnett, V . Filkov, and P. Devanbu, “A large scale study of programming languages and code quality in github,” in Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, 2014, pp. 155–165

    work page 2014

  4. [4]

    Understanding language selection in multi-language software projects on github,

    W. Li, N. Meng, L. Li, and H. Cai, “Understanding language selection in multi-language software projects on github,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2021, pp. 256–257

    work page 2021

  5. [5]

    On the impact of multi-language development in machine learning frameworks,

    M. Grichi, E. E. Eghan, and B. Adams, “On the impact of multi-language development in machine learning frameworks,” in 2020 IEEE Interna- tional Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2020, pp. 546–556

    work page 2020

  6. [6]

    An exploratory study on just- in-time multi-programming-language bug prediction,

    Z. Li, J. Ji, P. Liang, R. Mo, and H. Liu, “An exploratory study on just- in-time multi-programming-language bug prediction,” Information and Software Technology, vol. 175, p. 107524, 2024

    work page 2024

  7. [7]

    Understanding resolu- tion of multi-language bugs: An empirical study on apache projects,

    Z. Li, W. Wang, S. Wang, P. Liang, and R. Mo, “Understanding resolu- tion of multi-language bugs: An empirical study on apache projects,” in 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . IEEE, 2023, pp. 1–11

    work page 2023

  8. [8]

    Declarative static analysis for multilingual programs using codeql,

    D. Youn, S. Lee, and S. Ryu, “Declarative static analysis for multilingual programs using codeql,” Software: Practice and Experience , vol. 53, no. 7, pp. 1472–1495, 2023

    work page 2023

Show all 52 references
  1. [9]

    Multi-language software development: Issues, challenges, and solutions,

    H. Yang, Y . Nong, S. Wang, and H. Cai, “Multi-language software development: Issues, challenges, and solutions,” IEEE Transactions on Software Engineering, vol. 50, no. 3, pp. 512–533, 2024

    2024

  2. [10]

    Learning to detect and localize multilingual bugs,

    H. Yang, Y . Nong, T. Zhang, X. Luo, and H. Cai, “Learning to detect and localize multilingual bugs,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 2190–2213, 2024

    2024

  3. [11]

    Towards understanding and reasoning about android interoperations,

    S. Bae, S. Lee, and S. Ryu, “Towards understanding and reasoning about android interoperations,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 2019, pp. 223– 233

    2019

  4. [12]

    On multi-language software development, cross-language links and accompanying tools: a survey of professional software developers,

    P. Mayer, M. Kirsch, and M. A. Le, “On multi-language software development, cross-language links and accompanying tools: a survey of professional software developers,” Journal of Software Engineering Research and Development , vol. 5, pp. 1–33, 2017

    2017

  5. [13]

    Mathprompter: Mathematical reasoning using large language models,

    S. Imani, L. Du, and H. Shrivastava, “Mathprompter: Mathematical reasoning using large language models,” in ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models

    2023

  6. [14]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei et al. , “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  7. [15]

    From llms to llm- based agents for software engineering: A survey of current, challenges and future,

    H. Jin, L. Huang, H. Cai, J. Yan, B. Li, and H. Chen, “From llms to llm- based agents for software engineering: A survey of current, challenges and future,” arXiv preprint arXiv:2408.02479 , 2024

    2024 arXiv

  8. [16]

    Multi-language software development in the llm era: Insights from practitioners’ conversations with chatgpt,

    L. Aguiar, M. Paixao, R. Carmo, E. Soares, A. Leal, M. Freitas, and E. Gama, “Multi-language software development in the llm era: Insights from practitioners’ conversations with chatgpt,” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Enginee...

    2024

  9. [17]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  10. [18]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  11. [19]

    Javabench: A benchmark of object-oriented code generation for evaluating large language models,

    J. Cao, Z. Chen, J. Wu, S.-C. Cheung, and C. Xu, “Javabench: A benchmark of object-oriented code generation for evaluating large language models,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , 2024, pp. 870–882

    2024

  12. [20]

    Evaluating large language models in class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Evaluating large language models in class-level code generation,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  13. [21]

    Escalating llm-based code translation benchmarking into the class-level era,

    P. Xue, L. Wu, C. Wang, X. Li, Z. Yang, R. Jin, Y . Zhang, J. Li, Y . Pei, Z. Shen et al., “Escalating llm-based code translation benchmarking into the class-level era,” arXiv preprint arXiv:2411.06145 , 2024

    2024 arXiv

  14. [22]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,

    K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

    2024

  15. [23]

    Automated benchmark generation for repository-level coding tasks,

    K. Vergopoulos, M. N. Mueller, and M. Vechev, “Automated benchmark generation for repository-level coding tasks,” in ICLR 2025 Third Workshop on Deep Learning for Code

    2025

  16. [24]

    Repobench: Benchmarking repository- level code auto-completion systems,

    T. Liu, C. Xu, and J. McAuley, “Repobench: Benchmarking repository- level code auto-completion systems,” arXiv preprint arXiv:2306.03091 , 2023

    2023 arXiv

  17. [25]

    Llms in web development: Evaluat- ing llm-generated php code unveiling vulnerabilities and limitations,

    R. T ´oth, T. Bisztray, and L. Erd˝odi, “Llms in web development: Evaluat- ing llm-generated php code unveiling vulnerabilities and limitations,” in International Conference on Computer Safety, Reliability, and Security . Springer, 2024, pp. 425–437

    2024

  18. [26]

    Web-bench: A llm code benchmark based on web standards and frameworks,

    K. Xu, Y . Mao, X. Guan, and Z. Feng, “Web-bench: A llm code benchmark based on web standards and frameworks,” arXiv preprint arXiv:2505.07473, 2025

    2025 arXiv

  19. [27]

    Webapp1k: A practical code-generation benchmark for web app development,

    Y . Cui, “Webapp1k: A practical code-generation benchmark for web app development,” arXiv preprint arXiv:2408.00019 , 2024

    2024 arXiv

  20. [28]

    Da-code: Agent data science code generation benchmark for large language models,

    Y . Huang, J. Luo, Y . Yu, Y . Zhang, F. Lei, Y . Wei, S. He, L. Huang, X. Liu, J. Zhao et al. , “Da-code: Agent data science code generation benchmark for large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024,...

    2024

  21. [29]

    Benchmarking data science agents,

    Y . Zhang, Q. Jiang, X. XingyuHan, N. Chen, Y . Yang, and K. Ren, “Benchmarking data science agents,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 5677–5700

    2024

  22. [30]

    Llms for science: Usage for code generation and data analysis,

    M. Nejjar, L. Zacharias, F. Stiehle, and I. Weber, “Llms for science: Usage for code generation and data analysis,” Journal of Software: Evolution and Process , vol. 37, no. 1, p. e2723, 2025

    2025

  23. [31]

    Exploring and characterizing large language models for embedded system development and debugging,

    Z. Englhardt, R. Li, D. Nissanka, Z. Zhang, G. Narayanswamy, J. Breda, X. Liu, S. Patel, and V . Iyer, “Exploring and characterizing large language models for embedded system development and debugging,” in Extended Abstracts of the CHI Conference on Human Factors in Computing ...

    2024

  24. [32]

    Deploying and evaluating llms to program service mobile robots,

    Z. Hu, F. Lucchetti, C. Schlesinger, Y . Saxena, A. Freeman, S. Modak, A. Guha, and J. Biswas, “Deploying and evaluating llms to program service mobile robots,” IEEE Robotics and Automation Letters , vol. 9, no. 3, pp. 2853–2860, 2024

    2024

  25. [33]

    Domaineval: An auto-constructed benchmark for multi-domain code generation,

    Q. Zhu, J. Cao, Y . Lu, H. Lin, X. Han, L. Sun, and S.-C. Cheung, “Domaineval: An auto-constructed benchmark for multi-domain code generation,” in Proceedings of the AAAI Conference on Artificial Intel- ligence, vol. 39, no. 24, 2025, pp. 26 148–26 156

    2025

  26. [34]

    Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,

    Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, L. Shen, Z. Wang, A. Wang, Y . Li et al. , “Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data ...

    2023

  27. [35]

    Multipl-e: a scalable and polyglot approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman et al., “Multipl-e: a scalable and polyglot approach to benchmarking neural code generation,” IEEE Transactions on Software Engineering , vol. 49, no. 7...

    2023

  28. [36]

    Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution,

    R. Xu, J. Cao, Y . Lu, H. Lin, X. Han, B. He, S.-C. Cheung, and L. Sun, “Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution,” arXiv preprint arXiv:2408.13001 , 2024

    2024 arXiv

  29. [37]

    {PolyCruise}: A {Cross- Language} dynamic information flow analysis,

    W. Li, J. Ming, X. Luo, and H. Cai, “ {PolyCruise}: A {Cross- Language} dynamic information flow analysis,” in 31st USENIX Se- curity Symposium (USENIX Security 22) , 2022, pp. 2513–2530

    2022

  30. [38]

    Polyfax: A toolkit for characterizing multi- language software,

    W. Li, L. Li, and H. Cai, “Polyfax: A toolkit for characterizing multi- language software,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1662–1666

    2022

  31. [39]

    Abstract syntax net- works for code generation and semantic parsing,

    M. Rabinovich, M. Stern, and D. Klein, “Abstract syntax net- works for code generation and semantic parsing,” arXiv preprint arXiv:1704.07535, 2017

    2017 arXiv

  32. [40]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

    2009 arXiv

  33. [41]

    Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization,

    Q. Peng, Y . Chai, and X. Li, “Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization,” in Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 20...

    2024

  34. [42]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  35. [43]

    Codebenchgen: Creating scalable execution-based code generation benchmarks,

    Y . Xie, A. Xie, D. Sheth, P. Liu, D. Fried, and C. Rose, “Codebenchgen: Creating scalable execution-based code generation benchmarks,” arXiv preprint arXiv:2404.00566, 2024

    2024 arXiv

  36. [44]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  37. [45]

    Glm-4 model api documentation,

    Z. AI, “Glm-4 model api documentation,” https://open.bigmodel.cn/dev/ api/normal-model/glm-4, 2024, accessed: May 19, 2025

    2024

  38. [46]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al. , “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  39. [47]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools,

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao et al. , “Chatglm: A family of large language models from glm-130b to glm-4 all tools,” arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  40. [48]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  41. [49]

    Gemma: Open models based on gemini research and technology, 2024,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Love et al., “Gemma: Open models based on gemini research and technology, 2024,” URL https://arxiv. org/abs/2403.08295, vol. 2, pp. 10–19

    2024 arXiv

  42. [50]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al. , “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  43. [51]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  44. [52]

    Codegemma: Open code models based on gemma,

    C. Team, H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley et al. , “Codegemma: Open code models based on gemma,” arXiv preprint arXiv:2406.11409, 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.