Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Existing code benchmarks overestimate LLM ability: obfuscating target code drops test pass rates by up to 62.5%.

desk verdict A reusable obfuscation-based benchmark with real TPR drops, but the paper's central claim that the drops measure removed training familiarity is undercut by its own cutoff control. read the letter →

arxiv 2412.08109 v2 pith:DZMQVBSM submitted 2024-12-11 cs.SE cs.AI

Yuanliang Zhang , Yifan Xie , Shanshan Li , Ke Liu , Chong Wang , Zhouyang Jia , Xiangbing Huang , Jie Song
show 6 more authors
Chaopeng Luo Zhizheng Zheng Rulin Xu Yitong Liu Si Zheng Xiangke Liao
This is my paper · ORCID
classification cs.SEcs.AI
keywords largelanguagemodelscodegenerationbenchmarkcontaminationobfuscationtrainingdataleakagetestpassratefamiliarityreal-worldCprojects
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current code-generation benchmarks give inflated estimates of what large language models can actually do, because the target code is often already present in the models' training data. To remove that familiarity, the authors build OBFUSEVAL, a benchmark that collects 1,354 test-covered functions from five mature C projects and applies three levels of obfuscation - symbol renaming, structural restructuring, and semantic rewriting - to prompts, code, and dependencies. Across four large language models, the average test pass rate falls by 15.3% to 62.5% after obfuscation, with the largest drop for combined symbol-plus-structure obfuscation. The authors conclude that models' apparent expertise on familiar tasks is partly a specialist-in-familiarity effect, and that obfuscated benchmarks reveal a truer picture of production readiness.

What carries the argument

The load-bearing mechanism is three-level code obfuscation applied to otherwise real, test-covered functions. Symbol obfuscation rewrites every meaningful identifier in the target function and its context, so the prompt no longer matches any training snippet. Structure obfuscation uses compiler intermediate representation to inline called functions, changing the code's organization and length. Semantic obfuscation rewrites code to a functionally equivalent but implementationally different form, which currently requires human effort. The evaluation machinery is a containerized execution platform that replaces the original function with the model's output, recompiles the project, and runs the official test suite; comparing test pass rates before and after obfuscation is the instrument that quantifies loss of familiarity.

What would settle it

Give expert human C programmers the original and obfuscated versions of the same functions and compare their success rates; if humans show a comparable drop on obfuscated tasks, then the obfuscation itself raises difficulty, which would undermine the paper's attribution of the LLM drop to lost familiarity.

Watch

Extended reading notes

Core claim

The authors' central claim is that benchmark scores are inflated by code leakage: functions drawn from open-source projects are likely to appear in LLM training corpora, so models can solve them by recall rather than by genuine code synthesis. To expose this, OBFUSEVAL takes functions that entered real projects after known training cutoffs, rewrites their descriptions, and applies three levels of obfuscation to the code and its context: symbol obfuscation replaces all meaningful identifiers, structure obfuscation inlines called functions to change execution organization, and semantic obfuscation re-implements logic in a different but equivalent way. Generated code is then compiled inside a containerized sandbox and exercised against the projects' official test suites. The measured test pass rate drops consistently across all four evaluated models after obfuscation, by an average of 15.3% for semantic obfuscation up to 62.5% for combined symbol-plus-structure obfuscation. The authors read this as evidence that the models rely on familiarity with original code, so obfuscation measures the models' real, transferable capability.

Load-bearing premise

The argument assumes that obfuscation changes the code without increasing its intrinsic difficulty, so that the measured drop in test pass rate can be attributed entirely to the loss of training-data familiarity rather than to harder prompts.

Editorial extensions

If this is right

  • Scores on existing code benchmarks should not be equated with production capability; a model can perform well on familiar functions yet fail on the same task once the code is obfuscated.
  • Benchmark builders can reuse existing code corpora by obfuscating them, rather than waiting for new code, which addresses the timeliness problem.
  • Future LLM generations should be evaluated on obfuscated versions of the same tasks to distinguish genuine learning from memorization of training data.
  • Even code that passes official tests can carry non-functional defects such as missing null checks and inefficient loops, so functional correctness alone is an incomplete measure of readiness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the familiarity explanation holds, contamination is not binary: code written after a training cutoff can still resemble training code through reused idioms, so routine obfuscation would harden any code benchmark against gradual saturation.
  • The same de-familiarization treatment could transfer to code repair, code review, or test-generation benchmarks, where target-code exposure also risks inflating measured ability.
  • One way to disentangle difficulty from familiarity would be to measure how quickly models adapt to obfuscated versions through fine-tuning; fast recovery would suggest the drop is about retrieval rather than reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that existing code-generation benchmarks overestimate LLM capability because target code may have been seen during pretraining, and proposes OBFUSEVAL, a benchmark of 1,354 C functions from five mature projects, with functions drawn from post-training-cutoff PRs and obfuscated at symbol, structure, and semantic levels. Four LLMs are evaluated on code generation and code completion tasks using official test suites in a Docker-based execution platform. The reported average test pass rate (TPR) decreases by 15.3% to 62.5% after obfuscation, and the paper concludes that obfuscation removes training-data familiarity and thereby reveals the LLMs' real capability.

Significance. If the causal interpretation were supported, the paper would make a valuable methodological contribution: obfuscation is a reusable way to refresh benchmarks, and the project-level execution platform plus official test suites is a solid evaluation harness. The empirical finding that LLM TPR on these real C projects is low even on the original code (average 21.1%) is itself useful. However, the central causal claim and the headline decrease ratios depend on the assumption that obfuscation does not materially increase task difficulty, which the manuscript does not establish; the internal evidence actually suggests otherwise.

major comments (3)
  1. [Section IV-A2 and Table IV] For gpt-3.5-turbo-1106 and gpt-4-turbo-1106, the manuscript states that training data were finalized before April 2023, while the raw functions were collected from PRs merged from May to December 2023, making the original code already unseen by these models. Table IV nevertheless shows substantial TPR drops under obfuscation for these models, e.g., redis GPT4-1106 goes from 17.6 on Original to 4.7 on Symbol+Structure, and libgit2 GPT4-1106 goes from 23.1 to 0.0. If the drop were specifically caused by removing training familiarity, it should be near zero on already-unseen code. The persistence of large drops on unseen code implies that obfuscation-induced difficulty contributes substantially, so the conclusion that existing benchmarks overestimate LLM capability does not directly follow. This is the load-bearing confound and requires a control analysis, such as comparing obfuscation drops on pre-cutoff versus post-cutoff code or using a difficulty-matched baseline.
  2. [Section V, Code complexity] The threats-to-validity section concedes that structure obfuscation inlines called functions and can increase code length and token count, which 'potentially affect[s] the generation capabilities of large models.' The statement that the authors 'did not intentionally increase the complexity' is not a control. Because symbol obfuscation also replaces all identifiers, making the prompt lexically novel, the observed TPR decrease could reflect increased prompt difficulty (longer context, unfamiliar names, inlined code) rather than removal of memorized target code. The manuscript needs a quantitative analysis of prompt/code complexity before and after obfuscation and/or a control condition to separate difficulty from familiarity.
  3. [Section III-B3 and Table III] Semantic obfuscation is manual, applied to only 159 functions (106 from redis, 17 from libvips, 15 from lvgl, 10 from libgit2, and 11 from fluent), and Section IV-B2 admits that the semantic strategy is 'relatively simple.' Because the per-strategy and combination results are computed on different, small subsets rather than on a common benchmark, the aggregate comparison of strategy effectiveness and the headline 'up to 62.5%' is not a representative estimate across the dataset. The authors should report results on matched subsets and include confidence intervals, especially for the semantic and combination conditions.
minor comments (5)
  1. [Throughout] The text contains formatting artifacts such as 'O BFUS EVAL' and '/user-edit' before the RQ conclusions; these should be cleaned before publication.
  2. [Table IV] Table IV uses 'libgits' while the repository name and Table II use 'libgit2'; please make the labels consistent.
  3. [Fig. 7] Figure 7 contains code typos ('Void' for 'void') and prose typos ('Differenet'); the figure and caption should be proofread.
  4. [Section IV-A3] The paper defines pass@5 but does not state how many samples were generated per function or the sampling temperature; please specify the sampling setup for reproducibility.
  5. [Sections IV-B2 and V] The paper says 'we did not deliberately increase the difficulty' in Section IV-B2 and Section V, but Section III-B2 describes structure obfuscation as unfolding functions, which the authors concede may increase token count; please reconcile these statements with a quantitative measure of prompt length before and after obfuscation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the TPR decrease is an empirical measurement evaluated against external official test suites, not a quantity defined by fitted inputs or self-citations.

full rationale

The paper's central result is an observed decrease in test pass rate after code obfuscation, measured by replacing generated code into real projects and running their official test suites. This is an external benchmark-based measurement rather than a derived quantity that reduces to its input by construction. No parameter is fitted to the target result and then reported as a prediction; the 'decrease ratio' is read directly from pass-rate comparisons in Tables IV and V. The paper's interpretive claim that the decrease reflects removal of training-data familiarity is a causal hypothesis about why the drop occurs, not a definitional identity. The main weakness is that obfuscation (especially structure inlining) may increase task difficulty independently of familiarity, and Section V concedes this possibility; however, a confound between familiarity removal and increased difficulty is a threats-to-validity concern, not circularity. The paper's self-citations appear only in background discussion and are not load-bearing for the benchmark construction or the evaluation result. No equation, fitted parameter, or self-citation chain makes the conclusion equivalent to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark construction involves hand-chosen thresholds (time cutoff, star counts) and relies on several domain assumptions about LLM training exposure, obfuscation preserving semantics, and official tests as a correctness oracle. No new physical or methodological entities are postulated.

free parameters (1)
  • PR merge time cutoff (May 2023) = May 2023
    Hand-chosen cutoff to align with training data cutoffs of gpt-3.5-turbo-1106 and gpt-4-turbo-1106; determines which functions are treated as unseen and is central to the familiarity claim.
assumptions (4)
  • domain assumption Obfuscation preserves functionality and output while reducing model familiarity.
    Section III-B states this goal, and Section V acknowledges complexity was not intentionally increased, but no evidence is provided that the obfuscated prompts are not simply harder.
  • domain assumption Official test suites provide a valid functional correctness signal for generated code.
    Section III-C uses system testing with official suites; this is a standard assumption in code generation benchmarks but remains unvalidated for measuring 'real capability'.
  • domain assumption Pass@5 with default sampling is a stable estimator of model capability.
    Section IV-A3 states CPR/TPR use pass@5, but no variance, sample count, temperature, or estimator details are reported.
  • domain assumption PRs merged after the training cutoff remain unseen despite code clones and reuse.
    Section III-A1 and Gap 2 acknowledge clones may leak; the assumption is central to treating the original data as 'unseen'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar." pith.science (2026). https://pith.science/paper/DZMQVBSM

@misc{pith2026241208109,
  author       = {Pith},
  title        = {Pith review of: Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZMQVBSM}},
  note         = {Machine review of arXiv:2412.08109}
}
read the original abstract

Recently, large language models (LLMs) have shown strong potential in code generation tasks. However, there are still gaps before they can be fully applied in actual software development processes. Accurately assessing the code generation capabilities of large language models has become an important basis for evaluating and improving the models. Some existing works have constructed datasets to evaluate the capabilities of these models. However, the current evaluation process may encounter the illusion of "Specialist in Familiarity", primarily due to three gaps: the exposure of target code, case timeliness, and dependency availability. The fundamental reason for these gaps is that the code in current datasets may have been extensively exposed and exercised during the training phase, and due to the continuous training and development of LLM, their timeliness has been severely compromised. The key to solve the problem is to, as much as possible, evaluate the LLMs using code that they have not encountered before. Thus, the fundamental idea in this paper is to draw on the concept of code obfuscation, changing code at different levels while ensuring the functionality and output. To this end, we build a code-obfuscation based benchmark OBFUSEVAL. We first collect 1,354 raw cases from five real-world projects, including function description and code. Then we use three-level strategy (symbol, structure and semantic) to obfuscate descriptions, code and context dependencies. We evaluate four LLMs on OBFU- SEVAL and compared the effectiveness of different obfuscation strategy. We use official test suites of these projects to evaluate the generated code. The results show that after obfuscation, the average decrease ratio of test pass rate can up to 62.5%.

Figures

Figures reproduced from arXiv: 2412.08109 by the authors.

Figure 1
Figure 1. The familiarity level of code to LLM. The more [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of code obfuscation. by these benchmark tests does not match the distribution of code lengths in Humaneval, this comparison is inherently unfair. It therefore does not accurately reflect the validity of the benchmark tests. C. Motivation of Using Code Obfuscation As LLMs are continuously trained and released, traditional datasets would constantly be learned and trained by these models. Therefore, the timeli… view at source ↗
Figure 3
Figure 3. Workflow of dataset construction and testing [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An example of code obfuscation process cation methods, we designed an automated code execution and verification platform. The platform is built on Docker images, providing an isolated sandbox environment to ensure that the tested codes do not interfere with each other.…
Figure 5
Figure 5. Figure 5: Example of prompt for code generation scenarios [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: TPR under different code obfuscation strategies. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Examples of non-functional code quality issue. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 27 canonical work pages

  1. [1]

    An empirical study of the code generation of safety-critical software using llms,

    M. Liu, J. Wang, T. Lin, Q. Ma, Z. Fang, and Y . Wu, “An empirical study of the code generation of safety-critical software using llms,” Applied Sciences, vol. 14, no. 3, p. 1046, 2024

    2024

  2. [2]

    A review on code generation with llms: Application and evaluation,

    J. Wang and Y . Chen, “A review on code generation with llms: Application and evaluation,” in 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI) . IEEE, 2023, pp. 284–289

    2023

  3. [3]

    When llm-based code generation meets the software development process,

    F. Lin, D. J. Kim et al. , “When llm-based code generation meets the software development process,” arXiv preprint arXiv:2403.15852, 2024

    arXiv 2024

  4. [4]

    How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment,

    M. Kazemitabaar, X. Hou, A. Henley, B. J. Ericson, D. Weintrop, and T. Grossman, “How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment,” in Proceedings of the 23rd Koli Calling International Conference on Computing Education Research, 2023, pp. 1–12

    2023

  5. [5]

    Llm-based and retrieval-augmented control code generation,

    H. Koziolek, S. Gr ¨uner, R. Hark, V . Ashiwal, S. Linsbauer, and N. Es- kandani, “Llm-based and retrieval-augmented control code generation,” in Proc. 1st Int. Workshop on Large Language Models for Coffice (LLM4Code) at ICSE , vol. 2024, 2024

    work page 2024

  6. [6]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” arXiv preprint arXiv:2305.01210 , 2023

    arXiv 2023

  7. [7]

    Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,

    C. S. Xia, Y . Deng, and L. Zhang, “Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,” arXiv preprint arXiv:2403.19114 , 2024

    arXiv 2024

  8. [8]

    Evalu- ating language models for efficient code generation,

    J. Liu, S. Xie, J. Wang, Y . Wei, Y . Ding, and L. Zhang, “Evalu- ating language models for efficient code generation,” arXiv preprint arXiv:2408.06450, 2024

    arXiv 2024

Show all 64 references
  1. [9]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  2. [10]

    Multipl-e: A scalable and extensible approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman et al., “Multipl-e: A scalable and extensible approach to benchmarking neural code generation,” arXiv preprint arXiv:2208.08227 , 2022

    2022 arXiv

  3. [11]

    Aixbench: A code generation benchmark dataset,

    Y . Hao, G. Li, Y . Liu, X. Miao, H. Zong, S. Jiang, Y . Liu, and H. Wei, “Aixbench: A code generation benchmark dataset,” arXiv preprint arXiv:2206.13179, 2022

    2022 arXiv

  4. [12]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.- t. Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” in International Conference on Machine Learning . PMLR, 2023, pp. 18 319–18 345

    2023

  5. [13]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

    2024

  6. [14]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770 , 2023

    2023 arXiv

  7. [15]

    Deveval: Evaluating code generation in practical software projects,

    J. Li, G. Li, Y . Zhao, Y . Li, Z. Jin, H. Zhu, H. Liu, K. Liu, L. Wang, Z. Fang et al., “Deveval: Evaluating code generation in practical software projects,” arXiv preprint arXiv:2401.06401 , 2024

    2024 arXiv

  8. [16]

    Domaineval: An auto-constructed benchmark for multi-domain code generation,

    Q. Zhu, J. Cao, Y . Lu, H. Lin, X. Han, L. Sun, and S.-C. Cheung, “Domaineval: An auto-constructed benchmark for multi-domain code generation,” arXiv preprint arXiv:2408.13204 , 2024

    2024 arXiv

  9. [17]

    “Github,” https://github.com/, 2024

    2024

  10. [18]

    Automatic code generation from design patterns,

    F. J. Budinsky, M. A. Finnie, J. M. Vlissides, and P. S. Yu, “Automatic code generation from design patterns,” IBM systems Journal , vol. 35, no. 2, pp. 151–171, 1996

    1996

  11. [19]

    Intellicode compose: Code generation using transformer,

    A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 1433–1443

    2020

  12. [20]

    Towards better multilingual code search through cross-lingual con- trastive learning,

    X. Huang, Y . Ma, H. Zhou, Z. Jiang, Y . Zhang, T. Wang, and S. Li, “Towards better multilingual code search through cross-lingual con- trastive learning,” in Proceedings of the 14th Asia-Pacific Symposium on Internetware, 2023, pp. 22–32

    2023

  13. [21]

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,

    M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  14. [22]

    gpt-4 technical report,

    Openai, “gpt-4 technical report, ” corr, vol. abs/2303.08774, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  15. [23]

    Glm: General language model pretraining with autoregressive blank infilling,

    Z. Du, Y . Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model pretraining with autoregressive blank infilling,” arXiv preprint arXiv:2103.10360 , 2021

    2021 arXiv

  16. [24]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

    2022 arXiv

  17. [25]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y . Wang et al. , “A survey on evaluation of large language models,” arXiv preprint arXiv:2307.03109 , 2023

    2023 arXiv

  18. [26]

    Pangu-coder2: Boosting large language models for code with ranking feedback,

    B. Shen, J. Zhang, T. Chen, D. Zan, B. Geng, A. Fu, M. Zeng, A. Yu, J. Ji, J. Zhao et al., “Pangu-coder2: Boosting large language models for code with ranking feedback,” arXiv preprint arXiv:2307.14936 , 2023

    2023 arXiv

  19. [27]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei et al. , “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  20. [28]

    Large language models meet nl2code: A survey,

    D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, W. Yongji, and J.-G. Lou, “Large language models meet nl2code: A survey,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023, pp. 7443–7464

    2023

  21. [29]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” arXiv preprint arXiv:2306.08568 , 2023

    2023 arXiv

  22. [30]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  23. [31]

    At which training stage does code data help llms reasoning?

    Y . Ma, Y . Liu, Y . Yu, Y . Zhang, Y . Jiang, C. Wang, and S. Li, “At which training stage does code data help llms reasoning?” arXiv preprint arXiv:2309.16298, 2023

    2023 arXiv

  24. [32]

    Cct5: A code- change-oriented pre-trained model,

    B. Lin, S. Wang, Z. Liu, Y . Liu, X. Xia, and X. Mao, “Cct5: A code- change-oriented pre-trained model,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2023, pp. 1509–1521

    2023

  25. [33]

    Codep: grammatical seq2seq model for general-purpose code generation,

    Y . Dong, G. Li, and Z. Jin, “Codep: grammatical seq2seq model for general-purpose code generation,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 188–198

    2023

  26. [34]

    Incoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  27. [35]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    2023 arXiv

  28. [36]

    (2023) instruct-starcoder

    “(2023) instruct-starcoder.” https://huggingface.co/ GeorgiaTechResearchInstitute/starcoder-gpteacher-code-instruct, 2023

    2023

  29. [37]

    Chatgpt,

    “Chatgpt,” https://chatgpt.com, 2024

    2024

  30. [38]

    Skcoder: A sketch- based approach for automatic code generation,

    J. Li, Y . Li, G. Li, Z. Jin, Y . Hao, and X. Hu, “Skcoder: A sketch- based approach for automatic code generation,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2124–2135

    2023

  31. [39]

    Karecoder: A new knowledge-enriched code generation system,

    T. Huang, Z. Sun, Z. Jin, G. Li, and C. Lyu, “Karecoder: A new knowledge-enriched code generation system,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineer- ing: Companion Proceedings , 2024, pp. 270–271

    2024

  32. [40]

    Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,

    J. Li, G. Li, X. Zhang, Y . Dong, and Z. Jin, “Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,” arXiv preprint arXiv:2404.00599 , 2024

    2024 arXiv

  33. [41]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  34. [42]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x,

    Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, Z. Wang, L. Shen, A. Wang, Y . Li et al. , “Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x,” arXiv preprint arXiv:2303.17568, 2023

    2023 arXiv

  35. [43]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al. , “Competition- level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  36. [44]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,

    T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman et al., “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” arXiv preprint arXiv:1809.08887 , 2018

    2018 arXiv

  37. [45]

    Beyond code generation: Assessing code llm maturity with postconditions,

    F. He, J. Zhai, and M. Pan, “Beyond code generation: Assessing code llm maturity with postconditions,” arXiv preprint arXiv:2407.14118 , 2024

    2024 arXiv

  38. [46]

    Ex- ploring and evaluating hallucinations in llm-powered code generation,

    F. Liu, Y . Liu, L. Shi, H. Huang, R. Wang, Z. Yang, and L. Zhang, “Ex- ploring and evaluating hallucinations in llm-powered code generation,” arXiv preprint arXiv:2404.00971 , 2024

    2024

  39. [47]

    Calibration and correctness of language models for code,

    C. Spiess, D. Gros, K. S. Pai, M. Pradel, M. R. I. Rabin, A. Alipour, S. Jha, P. Devanbu, and T. Ahmed, “Calibration and correctness of language models for code,” arXiv preprint arXiv:2402.02047 , 2024

    2024 arXiv

  40. [48]

    Benchmarking the communication com- petence of code generation for llms and llm agent,

    J. J. Wu and F. H. Fard, “Benchmarking the communication com- petence of code generation for llms and llm agent,” arXiv preprint arXiv:2406.00215, 2024

    2024 arXiv

  41. [49]

    Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,” arXiv preprint arXiv:2308.01861, 2023

    2023 arXiv

  42. [50]

    Acecoder: An effective prompting technique specialized in code generation,

    J. Li, Y . Zhao, Y . Li, G. Li, and Z. Jin, “Acecoder: An effective prompting technique specialized in code generation,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  43. [51]

    Leetcode,

    “Leetcode,” https://leetcode.cn/, 2024

    2024

  44. [52]

    Code obfuscation against static and dynamic reverse engineering,

    S. Schrittwieser and S. Katzenbeisser, “Code obfuscation against static and dynamic reverse engineering,” in Information Hiding: 13th Interna- tional Conference, IH 2011, Prague, Czech Republic, May 18-20, 2011, Revised Selected Papers 13 . Springer, 2011, pp. 270–284

    2011

  45. [53]

    Protecting software through obfuscation: Can it keep pace with progress in code analysis?

    S. Schrittwieser, S. Katzenbeisser, J. Kinder, G. Merzdovnik, and E. Weippl, “Protecting software through obfuscation: Can it keep pace with progress in code analysis?” Acm computing surveys (csur), vol. 49, no. 1, pp. 1–37, 2016

    2016

  46. [54]

    Code obfuscation literature survey,

    A. Balakrishnan and C. Schulze, “Code obfuscation literature survey,” CS701 Construction of compilers , vol. 19, p. 31, 2005

    2005

  47. [55]

    “Redis,” https://github.com/redis/redis, 2024

    2024

  48. [56]

    Libgit2,

    “Libgit2,” https://github.com/libgit2/libgit2, 2024

    2024

  49. [57]

    Libvips,

    “Libvips,” https://github.com/libvips/libvips, 2024

    2024

  50. [58]

    “Fluent,” https://github.com/fluent/fluent-bit, 2024

    2024

  51. [59]

    “lvgl,” https://github.com/lvgl/lvgl, 2024

    2024

  52. [60]

    “Nltk,” https://www.nltk.org/, 2024

    2024

  53. [61]

    “Llvm,” https://llvm.org/, 2024

    2024

  54. [62]

    gpt-3.5-turbo,

    “gpt-3.5-turbo,” https://platform.openai.com/docs/models/gpt-3-5-turbo, 2024

    2024

  55. [63]

    gpt-4.0-turbo,

    “gpt-4.0-turbo,” https://platform.openai.com/docs/models/gpt-4o, 2024

    2024

  56. [64]

    Deepseek-coder-v2,

    “Deepseek-coder-v2,” https://github.com/deepseek-ai/DeepSeek-Coder- V2, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.