Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

CODEMENV: Benchmarking Large Language Models on Code Migration

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CODEMENV, a 922-example benchmark for cross-version code migration, reports that the best of nine LLMs reaches 43.84% pass@1 on easy legacy-to-new migration and that models are far weaker migrating new code to old environments.

desk verdict A genuinely useful first benchmark for cross-version code migration, but the headline numbers rest on a three-test-case oracle and a confounded OLD2NEW vs NEW2OLD comparison. read the letter →

arxiv 2506.00894 v1 pith:MDWOPUFD submitted 2025-06-01 cs.SE cs.AIcs.CLcs.LG

Keyuan Cheng , Xudong Shen , Yihao Yang , Tengyue Wang , Yang Cao , Muhammad Asif Ali , Hanbin Wang , Lijie Hu
show 1 more author
Di Wang
This is my paper
classification cs.SEcs.AIcs.CLcs.LG
keywords codemigrationcross-versioncompatibilitylargelanguagemodelsbenchmarkAPIchangesPythonJavapass@k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CODEMENV is a new benchmark that tests whether large language models can migrate code across library and JDK versions, built from 922 real function changes across 19 Python and Java packages. The paper reports that the best of nine evaluated models, GPT-4O, reaches 43.84% pass@1 on the easy legacy-to-new migration subset, that the reported average pass@1 across models is 26.50%, and that every model struggles more when asked to move new code to an older environment. It also finds that models can often locate incompatible functions but are markedly worse at recalling exactly how those functions changed, and that the most common migration failure is still calling the incompatible function in the rewritten code. The benchmark matters because cross-version compatibility is a practical maintenance burden, and it gives that burden a concrete, repeatable measurement.

What carries the argument

The load-bearing object is the CODEMENV dataset and its three-task evaluation protocol. Function changes are collected from official documentation and release notes, classified as addition, deprecation, or replacement, and then used by GPT-4 to generate runnable example code plus exactly three test cases per function. Tasks 1 and 2 are scored by an agent-based comparator against ground-truth answers, while Task 3 (the migration task) is scored by whether the migrated code produces the same outputs as the original on the three test cases, with Pass@k measuring success over multiple attempts. The OLD2NEW versus NEW2OLD split is the main controlled variable: for replacement-type changes the same function change is tested in both directions.

What would settle it

Re-run Task-3 with a larger, independently built test suite (e.g., 20 cases per function including edge and randomized inputs) and recompute pass@1; if a material share of migrations that passed the original three cases now produce different outputs, the reported rates overstate migration ability.

Watch

Extended reading notes

Core claim

The paper's central claim is that cross-version code migration can be benchmarked systematically, and that current LLMs are far from reliable at it. CODEMENV splits migration into three abilities: locating version-incompatible functions, describing the change (type, version, and replacement function), and rewriting the code so it behaves identically in the target environment. On the migration task the models are consistently better in the OLD2NEW direction than in NEW2OLD, which the authors attribute to greater familiarity with newer function versions in training data. The benchmark is built by collecting function changes from official release notes, using GPT-4 to generate original code and three test cases per example, and treating identical outputs on those three test cases as correctness for migrated code.

Load-bearing premise

The migration score depends on exactly three test cases that GPT-4 wrote and validated; if those three cases do not exercise the full behavior of the function, incomplete or incorrect migrations can be counted as correct.

Editorial extensions

If this is right

  • Legacy-to-new migration is the right first target for automated tools: GPT-4O scores 43.84% pass@1 on easy OLD2NEW but only 31.60% on easy NEW2OLD.
  • Spending more generation attempts helps on easy migrations, but on hard NEW2OLD cases Pass@5 is barely above Pass@1, so sampling alone will not fix reverse migration.
  • Diagnosis and repair are distinct skills: GPT-TURBO-3.5 leads Task-1 locating but is weak on Task-3 migration, so end-to-end correctness has to be measured directly.
  • The most common failure mode, calling the incompatible function anyway, means migration quality is gated by complete identification and replacement of every incompatible call.
  • Version-change recall is a bottleneck, with the average Task-2 accuracy for general LLMs at 33.96%, so models need better access to accurate change history.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Task-3 correctness rests on only three GPT-4-generated test cases, the reported pass rates are likely optimistic; independently written or property-based test suites would give a stricter and probably lower estimate.
  • The NEW2OLD gap looks like a training-data recency effect; a testable prediction is that models trained on time-balanced or historically expanded documentation would reduce it.
  • CODEMENV could be reused as an evaluation harness for API-knowledge editing or retrieval-augmented migration: run the same tasks with and without release notes supplied, and the difference separates stored knowledge from reasoning ability.
  • A natural next step is repository-level migration, where incompatible calls interact across files and build configuration, beyond the single-function edits this benchmark measures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents CODEMENV, a benchmark for evaluating LLMs on cross-version code migration across Python and Java. It contains 922 examples from 19 packages and defines three tasks: locating version-incompatible functions, describing the changes those functions underwent, and migrating code to a target environment. Nine LLMs are evaluated; the headline results are that GPT-4O reaches 43.84% pass@1 on the OLD2NEW easy migration subset and that the average pass@1 across models is 26.50%. The authors also report that models perform better when migrating old code to newer environments and identify a logical-inconsistency failure mode in which models cite irrelevant version changes. The dataset and prompts are publicly released.

Significance. If the evaluation methodology is valid, CODEMENV fills a genuine gap: existing code benchmarks focus on generation and cross-language translation, not cross-version migration within an environment. The three-task decomposition is useful, the public release of data and prompts is valuable, and the error analysis distinguishes call errors, run errors, and wrong answers. The paper also ships a concrete, falsifiable claim about model asymmetry between OLD2NEW and NEW2OLD migration. However, the strength of these claims is limited by the evaluation oracle for Task-3, which relies on only three GPT-generated test cases, and by the unvalidated LLM judge used for Tasks 1 and 2. The benchmark is potentially a solid contribution, but the validity of the reported numbers needs to be demonstrated rather than assumed.

major comments (3)
  1. [Section 3.4, Eq. (3); Section 3.5, Step 3] Task-3 correctness is gated on just three test cases generated and refined by GPT-4. Eq. (3) requires identical outputs on these three cases, but there is no coverage requirement ensuring that the three cases exercise the API change, and no static check that the migrated code no longer calls the version-incompatible function. A migration that leaves a deprecated-but-runnable call in place, or that changes behavior only on inputs outside the three chosen cases, is scored as correct. Since the headline numbers (GPT-4O at 43.84% pass@1 on OLD2NEW easy, average 26.50%) are computed with this oracle, the central quantitative claim is only as strong as these three cases. Please increase the number of tests, add coverage or behavioral-difference checks, or validate the generated tests against a held-out set of inputs; at minimum, report how many samples would be reclassified with a larger test budget.
  2. [Section 3.4, agent-based evaluation; Appendix B, Prompt 3] Tasks 1 and 2 are scored by an LLM judge using the instruction to 'loosely compare' the predicted change with the reference and to accept answers that are 'related or only have a little difference.' No validation of this judge against human labels or exact-match baselines is reported. This creates a risk that Task-2 accuracies, such as DEEPSEEK-V3's 42.06% average, are inflated by lenient judging. Please report judge agreement with human annotations on a sample, or replace the loose comparison with a more constrained scoring procedure.
  3. [Section 3.5, Steps 2-3; Section 4.2, Table 3] GPT-4 generated the original code and the test cases, and GPT-4O is also the best-performing model on the resulting benchmark. This is not circular in the technical sense, but it creates a contamination risk: GPT-4O may have memorized the generated examples or their stylistic patterns. Please report whether the models' training cutoffs predate the dataset, run a simple contamination probe (e.g., asking GPT-4O to reproduce a sample's code from the problem description), or evaluate on a held-out set generated after the models' release.
minor comments (4)
  1. [Abstract vs. Section 4.1] The abstract says the evaluation uses 'seven LLMs,' but Section 4.1 and Table 3 report results for nine models; please make the count consistent.
  2. [Section 4.2, Figure 4] The text mentions 'DEEPSEEK-CHAT' in the RunError discussion, but the evaluated model is DEEPSEEK-V3; please correct the name.
  3. [Section 3.2 and Section 4.1] The notation for metrics is inconsistent: Section 3.4 uses AccTask-1, AccTask-2, AccTask-3, while Section 4.1 writes Acc Task_1, Acc Task_2, Acc Task_3; unify the notation.
  4. [Throughout] There are several typographical errors: 'Answring' and 'imcompatible' in Figure 2, 'Env ironments' in Section 3, and the duplicate 'Prompt 5' heading in Appendix B; a copyedit pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CODEMENV's evaluation chain is not defined by the models' outputs or by self-citation.

full rationale

CODEMENV's construction and scoring chain is self-contained and contains no fitting-to-prediction loop. Task-3 correctness (Eq. 3) compares migrated outputs to original outputs on three test cases; the test cases are generated and refined by GPT-4 in Step 3 of Section 3.5, but the score is not defined in terms of the evaluated models' outputs, and no parameter is fitted to make any model's result come out a particular way. The use of GPT-4 to generate code and tests and also for the agent-based judgment of Tasks 1 and 2 is a potential source of bias or reduced validity, but it is not circularity by the paper's own equations: the ground truth is taken from official version-release documentation, and the judge is asked to compare against that ground truth rather than to produce it. The paper's self-citations (Cheng et al. 2024; Zhang et al. 2024) occur only in the knowledge-editing related-work section and are not load-bearing for the benchmark's construction or results. The Limitations section acknowledges the small Java set and the difficulty of rigorous Java unit tests, which is a validity concern rather than a circular derivation. Accordingly, no specific circular step can be quoted with a reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on domain assumptions about the quality of release notes, the representativeness of GPT-4-generated code, the sufficiency of three test cases, and the reliability of an LLM judge. There are no free mathematical constants or fitted parameters in the sense of a scientific derivation, but the scoring thresholds and test counts are hand-chosen and influence the results.

free parameters (3)
  • number_of_test_cases = 3
    Each code sample is evaluated against exactly three generated test cases (Eq. 3); the count is chosen by the authors, not derived from a power analysis.
  • version_tolerance = 0.5
    Task 2 accepts a predicted version as correct if it is within 0.5 of the ground-truth version (Eq. 2); the tolerance is an arbitrary scoring choice.
  • test_refinement_rounds = 3
    Test cases are refined at most three times before the sample is discarded (Section 3.5 Step 3); this cap is a design choice that affects dataset composition.
assumptions (4)
  • domain assumption Function changes cataloged from official release notes accurately represent all relevant incompatibilities between the selected versions.
    Central to Task 1 ground truth; if release notes are incomplete, the 'incompatible functions' reference answer is wrong. Invoked throughout Section 3.2 and 3.5.
  • domain assumption GPT-4-generated code snippets are representative of real code that requires migration.
    The benchmark's examples are not real-world code; the code is produced by prompting GPT-4 (Section 3.5, Step 2). If generated code is systematically simpler or different from actual code, performance on CODEMENV does not transfer to real migration.
  • domain assumption Three GPT-4-generated test cases are sufficient to detect functional equivalence.
    Task 3 correctness is defined by equality of outputs on exactly three test cases (Eq. 3). This is the load-bearing assumption for the migration accuracy numbers.
  • domain assumption An LLM judge performing a 'loose comparison' of change descriptions produces valid labels for Task 2.
    The agent-based evaluation of Task 2 uses a language model to judge whether the predicted function change matches the reference (Prompt 3). The validity of the Task 2 scores depends on this judge's reliability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CODEMENV: Benchmarking Large Language Models on Code Migration." pith.science (2026). https://pith.science/paper/MDWOPUFD

@misc{pith2026250600894,
  author       = {Pith},
  title        = {Pith review of: CODEMENV: Benchmarking Large Language Models on Code Migration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MDWOPUFD}},
  note         = {Machine review of arXiv:2506.00894}
}
read the original abstract

Large language models (LLMs) have shown remarkable capabilities across various software engineering tasks; however, their effectiveness in code migration, adapting code to run in different environments, remains insufficiently studied. In this work, we introduce CODEMENV: Code Migration Across Environment, a new benchmark specifically designed to assess LLMs' abilities in code migration scenarios. CODEMENV consists of 922 examples spanning 19 Python and Java packages, and covers three core tasks: (1) identifying functions incompatible with specific versions, (2) detecting changes in function definitions, and (3) adapting code to target environments. Experimental evaluation with seven LLMs on CODEMENV yields an average pass@1 rate of 26.50%, with GPT-4O achieving the highest score at 43.84%. Key findings include: (i) LLMs tend to be more proficient with newer function versions, which aids in migrating legacy code, and (ii) LLMs sometimes exhibit logical inconsistencies by identifying function changes irrelevant to the intended migration environment. The datasets are available at https://github.com/xdshen-ai/Benchmark-of-Code-Migration.

Figures

Figures reproduced from arXiv: 2506.00894 by the authors.

Figure 1
Figure 1. The function compare_chararrays un￾derwent changes after Numpy 1.26, creating compati￾bility issues between NumPy 1.26 and 2.0. tablished benchmarks. Beyond these general models, a range of specialized CodeLLMs have been introduced to further advance performance on code-related tasks. Notable examples include CodeT5+ (Wang et al., 2023), CodeLlama (Roz￾ière et al., 2023), and StarCoder2 (Lozhkov et al., 2024). Thank… view at source ↗
Figure 2
Figure 2. A data example of CODEMENV, which in￾cludes three tasks to evaluate LLMs on environment￾related programming skills. 3.1 Task Definition CODEMENV include the following three tasks: Task-1: Identify Version-Incompatible Func￾tions. Given a code snippet and a specified target environment version, the model is asked to pin￾point functions that are incompatible with that en￾vironment. This task is divided into two levels… view at source ↗
Figure 3
Figure 3. The construction process of CODEMENV. Step 1: We collect function change information and function descriptions from the official website; Step 2: Based on the collected functions, generate code that can run in the original version and its problem description; Step 3: Generate 3 test cases for each data and repeat three times until all cases can run correctly. tal of 587 samples, which are divided into two difficulty… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Error Analysis of Code Migration. CallError represents a function where an incompatible the en￾vironment is still called. RunError represents that the migrated code enters an infinite loop during execution. WrongAnswer represents this code runs normally and gets the re…
Figure 5
Figure 5. Figure 5: Case Study. We plot an example of NEW2OLD from Python (Easy) datasets and present the response of Task-1 and Task-2 for four LLMs. In this case study, we observe the phenomenon of logical inconsistency, where LLAMA-3.1-8B and GPT-TURBO-3.5 provide function changes that…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Compositional Architecture of Regret in Large Language Models

    cs.CL 2025-06 reject novelty 5.0 of 10

    The paper claims that regret in LLMs is encoded by interacting neuron groups detectable in the final hidden layer, using new S-CDI, RDS, and GIC metrics.

  2. Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images

    cs.AI 2025-06 reject novelty 4.0 of 10

    SHE lowers behavioral hallucination scores by about 10 percent by detecting low visual-textual similarity and projecting out the hallucinated direction in embedding space.

Reference graph

Works this paper leans on

39 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    https://api.semanticscholar.org/CorpusID:268232499 The claude 3 model family: Opus, sonnet, haiku

    work page

  4. [4]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Singh Behl, Alon Benhaim, Misha Bilenko, and Johan Bjorck. 2024. https://api.semanticscholar.org/CorpusID:269293048 Phi-3 technical report: A highly capable language model locally on your phone . ArXiv, abs...

    arXiv 2024

  5. [5]

    Abhinav Jauhri Abhimanyu Dubey et al. 2024. https://api.semanticscholar.org/CorpusID:271571434 The llama 3 herd of models . ArXiv, abs/2407.21783

    arXiv 2024

  6. [7]

    Aylton Almeida, Laerte Xavier, and Marco T \'u lio Valente. 2024. https://api.semanticscholar.org/CorpusID:272145979 Automatic library migration using large language models: First results . Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

    work page 2024

  7. [8]

    https://aws.amazon.com/q/developer/transform/?nc1=h_ls Amazon q developer: Transform code

    AmazonQ. https://aws.amazon.com/q/developer/transform/?nc1=h_ls Amazon q developer: Transform code . 2025

    work page 2025

  8. [9]

    Keyuan Cheng, Gang Lin, Haoyang Fei, Yuxuan Zhai, Lu Yu, Muhammad Asif Ali, Lijie Hu, and Di Wang. 2024. https://api.semanticscholar.org/CorpusID:268819534 Multi-hop question answering under temporal knowledge editing . ArXiv, abs/2404.00492

    arXiv 2024

Show all 39 references
  1. [10]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. https://api.semanticscholar.org/CorpusID:269128474 Evaluating large language models in class-level code generation . 2024 IEEE/ACM 46th Interna...

    2024

  2. [11]

    Hasan Ferit Eniser, Hanliang Zhang, Cristina David, Meng Wang, Maria Christakis, Brandon Paulsen, Joey Dodds, and Daniel Kroening. 2024. https://api.semanticscholar.org/CorpusID:269922036 Towards translating real-world code with llms: A study of translating to rust . ArXiv, ab...

    2024 arXiv

  3. [12]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. https://api.semanticscholar.org/CorpusID:211204736 Realm: Retrieval-augmented language model pre-training . ArXiv, abs/2002.08909

    2020 arXiv

  4. [13]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. 2021. https://api.semanticscholar.org/CorpusID:234790100 Measuring coding challenge competence with apps . Ar...

    2021 arXiv

  5. [14]

    Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister

    Cheng-Yu Hsieh, Sibei Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander J. Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. https://api.semanticscholar.org/CorpusID:260351459 Tool documentation enables zero-shot tool-usage with large language models . ArXiv, abs/2308.00675

    2023 arXiv

  6. [15]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Shanghaoran Quan, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. https://api.semanticscholar.org/CorpusID:27270...

    2024 arXiv

  7. [16]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. https://api.semanticscholar.org/CorpusID:270214176 A survey on large language models for code generation . ArXiv, abs/2406.00515

    2024 arXiv

  8. [17]

    Joe El Khoury. 2024. https://medium.com/@jelkhoury880/leveraging-large-language-models-for-automated-code-migration-and-repository-level-tasks-part-i-402fd892eef7 Leveraging large language models for automated code migration and repository-level tasks — part i

    2024

  9. [18]

    M \"u ller, and John Mylopoulos

    Kostas Kontogiannis, Johannes Martin, Kenny Wong, Richard Gregory, Hausi A. M \"u ller, and John Mylopoulos. 2010. https://api.semanticscholar.org/CorpusID:15226793 Code migration through transformations: an experience report . In Conference of the Centre for Advanced Studies ...

    2010

  10. [19]

    Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://api.semanticscholar.org/CorpusID:218869575 Retrieval-augmented gener...

    2020 arXiv

  11. [20]

    Xiaopeng Li, Shangwen Wang, Shasha Li, Jun Ma, Jie Yu, Xiaodong Liu, Jing Wang, Bing Ji, and Weimin Zhang. 2024. https://api.semanticscholar.org/CorpusID:273963883 Model editing for llms4code: How far are we? ArXiv, abs/2411.06638

    2024 arXiv

  12. [21]

    Zeyu Leo Liu, Shrey Pandit, Xi Ye, Eunsol Choi, and Greg Durrett. 2024. https://api.semanticscholar.org/CorpusID:271064726 Codeupdatearena: Benchmarking knowledge editing on api updates . ArXiv, abs/2407.06249

    2024 arXiv

  13. [22]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, and Qian Liu. 2024. https://api.semanticschola...

    2024 arXiv

  14. [23]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022 a . Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359--17372

    2022

  15. [24]

    Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2022 b . Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations

    2022

  16. [25]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, et al. 2024 a . http://arxiv.org/abs/2303.08774 Gpt-4 technical report

    2024 arXiv

  17. [26]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, et al

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, et al. 2024 b . http://arxiv.org/abs/2410.21276 Gpt-4o system card

    2024 arXiv

  18. [27]

    Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, and Andrew M

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, and Andrew M. 2024. https://api.semanticschola...

    2024 arXiv

  19. [28]

    Baptiste Rozi \`e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cris tian Cant \'o n Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexa...

    2023 arXiv

  20. [29]

    Zhihong Shao, Damai Dai, Daya Guo, Bo Liu (Benjamin Liu), Zihan Wang, and Huajian Xin. 2024. https://api.semanticscholar.org/CorpusID:269613809 Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model . ArXiv, abs/2405.04434

    2024 arXiv

  21. [30]

    Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, and Tao Yu. 2024. https://api.semanticscholar.org/CorpusID:267750919 Evor: Evolving retrieval for code generation . In Conference on Empirical Methods in Natural Language Processing

    2024

  22. [31]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  23. [32]

    Behrooz Omidvar Tehrani, Ishaani M, and Anmol Anubhai. 2024. https://api.semanticscholar.org/CorpusID:269743373 Evaluating human-ai partnership for llm-based code migration . Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

    2024

  24. [33]

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. https://api.semanticscholar.org/CorpusID:258685677 Codet5+: Open code large language models for code understanding and generation . In Conference on Empirical Methods in Natural ...

    2023

  25. [34]

    Joty, and Steven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. https://api.semanticscholar.org/CorpusID:237386541 Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation . ArXiv, abs/2109.00859

    2021 arXiv

  26. [35]

    Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. http://arxiv.org/abs/2303.10420 A comprehensive capability analysis of gpt-3 and gpt-3.5 series models

    2023 arXiv

  27. [36]

    Zhiqiang Yuan, Weitong Chen, Hanlin Wang, Kai Yu, Xin Peng, and Yiling Lou. 2024. https://api.semanticscholar.org/CorpusID:272988134 Transagent: An llm-based multi-agent system for code translation . ArXiv, abs/2409.19894

    2024 arXiv

  28. [37]

    Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijie Hu, and Di Wang. 2024. https://api.semanticscholar.org/CorpusID:273228000 Locate-then-edit for multi-hop factual recall under knowledge editing . ArXiv, abs/2410.06331

    2024 arXiv

  29. [38]

    Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795

    2023 arXiv

  30. [39]

    Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig

    Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. https://api.semanticscholar.org/CorpusID:252734952 Docprompting: Generating code by retrieving the docs . In International Conference on Learning Representations

    2022

  31. [40]

    Celal Ziftci, Stoyan Nikolov, Anna Sjovall, Bo Kim, Daniele Codecasa, and Max Kim. 2025. https://api.semanticscholar.org/CorpusID:277781333 Migrating code at scale with llms at google . ArXiv, abs/2504.09691

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.