Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Towards Advancing Code Generation with Large Language Models: A Research Roadmap

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that improving LLM code generation in real projects requires a six-layer architecture and a four-phase workflow that keeps developers actively involved, rather than relying on autonomous multi-agent pipelines.

desk verdict A plausible synthesis and roadmap for LLM code generation, but the central workflow recommendations are unvalidated and the layer/phase framing is internally inconsistent. read the letter →

arxiv 2501.11354 v1 pith:O3VINZWZ submitted 2025-01-20 cs.SE cs.AI

Haolin Jin , Huaming Chen , Qinghua Lu , Liming Zhu This is my paper · ORCID
classification cs.SEcs.AI
keywords largelanguagemodelscodegenerationLLM-basedagentssoftwareengineeringpromptbenchmarkssecurityhuman-in-the-loop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a research roadmap, not a new system. It argues that existing LLM code generation work can be understood through a six-layer architecture, from requirements through refinement, and that the field's next steps should follow a four-phase workflow: Input, Orchestration, Development, and Validation. The central message is that current evaluation rewards short, function-level, one-shot tasks, while real development needs class-level, multi-file tasks, security checks, and frequent human oversight. If the roadmap is right, future research should shift from autonomous black-box agents to a developer-in-the-loop model in which an orchestrator dynamically creates tasks and users validate understanding and output at each stage.

What carries the argument

The load-bearing object is the six-layer vision framework and its derived four-phase workflow. The six layers name the recurring components of LLM code generation: Requirements & User, Model Invocation, Planning & Reasoning, Synthesis, Execution/Verification/Validation, and Refinement & Debug. The four-phase workflow runs on two mechanisms: a clarity check in the Input phase that forces the model to confirm ambiguous requirements with the user, and an orchestrator in the Orchestration phase that performs dynamic task creation, spawning, merging, or reassigning agents based on execution results and human feedback, with agents stored in a model cloud. The workflow's point is to replace black-box autonomous generation with visible, incremental, human-monitored steps.

What would settle it

A controlled comparison on the same class-level, multi-file tasks would settle the roadmap's central recommendation: run the proposed developer-in-the-loop, clarity-checked workflow against an autonomous multi-agent baseline and measure wall-clock time, token cost, and final code quality. If the interactive workflow is slower and costlier with no quality gain, the core recommendation fails; a narrower test would compare pass rates on deliberately ambiguous tasks with and without the clarity-check step.

Watch

Extended reading notes

Core claim

The paper's central claim is that the apparent diversity of LLM code generation systems hides a common structure. It proposes six layers—requirements and user inputs, model invocation, planning and reasoning, synthesis, execution and verification, and refinement and debugging—that any approach can be mapped onto, and it derives a four-phase vision workflow from them. In this workflow, the model does not blindly generate from a prompt; it performs a clarity check with the user, an orchestrator dynamically creates and merges subtasks, development proceeds with user monitoring of each module, and validation combines system-level tests with human sign-off. The paper also catalogues four obstacles—prompt sensitivity, usability and token cost, code security, and evaluation gaps—and argues that the field should prioritize class-level and multi-file benchmarks, security testing, and explainable human-in-the-loop interaction over chasing higher scores on short benchmark tasks.

Load-bearing premise

The roadmap's practical value rests on the untested premise that frequent human-model interaction and an orchestrator's dynamic task creation improve reliability, explainability, and token efficiency without cancelling the productivity gains of automation; if the interaction overhead outweighs the gains, the recommended workflow loses its force.

Editorial extensions

If this is right

  • Agent frameworks should include an explicit clarity-check step, where the model asks the user for missing details before generating code.
  • Multi-agent systems should use an orchestrator that dynamically creates, merges, and reassigns tasks, reusing agents from a model cloud to control token consumption.
  • Development processes should keep a developer in the loop, reviewing and testing each generated module as it is written, so black-box failures surface early.
  • Evaluation should expand from function-level tasks to class-level and multi-file benchmarks with edge cases, plus readability and maintainability metrics.
  • Security of generated code should be tested at module level and treated as a first-class concern alongside functional correctness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The roadmap's own logic implies a measurable claim it does not test: the developer-in-the-loop workflow should beat an autonomous multi-agent baseline on time, token cost, and final code quality; a head-to-head study would turn the roadmap into a testable research program.
  • The six-layer architecture could be used as a completeness checklist for auditing any new LLM code-generation framework, an operationalization the paper leaves implicit.
  • The clarity-check mechanism points toward uncertainty-aware training: models that recognize gaps in their own knowledge and ask questions could be rewarded for doing so, extending the paper's suggestion into a training objective.
  • Class-level, multi-file benchmarks will probably need to be assembled from repository-level software engineering tasks with cross-module dependencies; the paper identifies the gap but stops short of designing such a benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a vision/roadmap for LLM-based code generation. The authors propose a six-layer architecture (Requirements & User, Model Invocation, Planning & Reasoning, Synthesis, Execution/Verification/Validation, Refinement & Debug) that categorizes components of existing LLM code generation systems, and a separate four-phase workflow (Input, Orchestration, Development, Validation) that emphasizes frequent human interaction, dynamic task creation, and an orchestrator-managed "model cloud." The paper then discusses technical challenges (prompt sensitivity, usability & consumption, code security) and evaluation challenges (benchmark design, metrics), offering suggestions such as clarity checks, developer-in-the-loop validation, and class-level multi-file benchmarks. The central contribution is prescriptive: researchers and tool builders should adopt the six-layer taxonomy and the proposed workflow to improve reliability, explainability, and token efficiency.

Significance. If the framework were rigorously established, it could provide a useful organizing structure for the rapidly growing literature on LLM code generation, and the suggested directions (class-level benchmarks, security evaluation, reproducibility practices) are timely and relevant. The paper deserves credit for making specific, falsifiable claims about what the community should do next and for grounding the taxonomy in a broad set of references. However, the significance is currently limited by the fact that the prescriptive core—especially the claimed benefits of frequent human interaction and dynamic task creation—is asserted without empirical or quasi-empirical support, and at least one cited reference points in the opposite direction. The taxonomy itself is a plausible descriptive synthesis, but its completeness and novelty are not established by any systematic method.

major comments (4)
  1. [Section 3.1.2] The sentence "Frequent developer-orchestrator interactions help reduce token cost while improving explainability and code comprehension by revealing the model's assumptions in real time" is a load-bearing empirical claim that is stated without evidence. The paper cites Xu et al. [41] earlier in the same section, which reports that developers using in-IDE code generation did not necessarily see large efficiency gains, and the text itself acknowledges that developers may invest as much effort in orchestrating workflows as in coding manually. The manuscript should either provide a user study, controlled experiment, or quantitative analysis supporting the claim, or explicitly reframe it as a hypothesis and discuss conditions under which the frequent-interaction approach could fail (e.g., interruption overhead, increased token cost of clarification, or loss of flow). As written, the roadmap's central recommendation is an assertion that the paper's own literature review undercuts.
  2. [Section 2.2 and Figure 2] The concepts of "Dynamic Task Creation" and "model cloud" are introduced as key components of the proposed workflow, but no concrete specification is given. For example, the text says "the orchestrator LLM performs system-level comprehension of the current task list to dynamically adjust the agents number based on task complexity," yet it does not define how task complexity is measured, how the decision to merge or spawn tasks is made, how the orchestrator resolves conflicting execution results, or how agents in the model cloud are selected and updated. Without a precise description (e.g., pseudocode, state machine, or explicit algorithm), the workflow is not actionable and cannot be implemented, evaluated, or compared against existing frameworks such as MetaGPT or ChatDev. The paper should also clarify the relationship between the six-layer architecture and the four-phase workflow; the abstract appears to conflate them, while Section 2.2 explicitly separates them.
  3. [Abstract and Section 2] The abstract states: "We propose a six-layer vision framework that categorizes code generation process into distinct phases, namely Input Phase, Orchestration Phase, Development Phase, and Validation Phase." This is inconsistent with Section 2.2, which says "Unlike the six-layer architecture, our vision does not aim to construct a concrete framework" and presents the four phases as a separate workflow that synthesizes insights from existing works. This inconsistency obscures the central contribution and forces the reader to determine whether the six layers map onto the four phases or are mutually independent artifacts. The authors should either provide an explicit mapping (e.g., a table showing which layers are active in which phases) or revise the abstract to describe the six-layer framework and the four-phase workflow as two distinct contributions.
  4. [Abstract and Section 3] The abstract claims the paper "systematically analyse[s]" the challenges, but Section 3 does not describe any methodology: there is no search strategy, inclusion/exclusion criteria, coding scheme, or procedure for deriving the taxonomy. The paper cites 46 references, which is a small and seemingly non-exhaustive sample of the LLM code generation literature (e.g., SWE-bench, CodeXGLUE, and recent agent benchmarks are not discussed). This matters because the completeness and representativeness of the six-layer framework is part of the paper's contribution. The authors should either remove "systematically" and describe the work as a perspective/roadmap based on selected prior work, or add a short methodology subsection stating how the reviewed works were selected and how the layer categories were derived.
minor comments (4)
  1. [Section 2.1] The subsection labels "Layer 1-3", "Layer 4-5", and "Layer 6" do not match the named layers in Figure 1; consider using the actual layer names (e.g., "Requirements & User, Model Invocation, and Planning & Reasoning Layers") to reduce cognitive load and avoid confusion.
  2. [Section 3.2] The sentence "Many existing works rely on simple metrics like Pass@K [46]" cites reference [46] (LDB debugger), but Pass@K is introduced in reference [3] (Chen et al., "Evaluating Large Language Models Trained on Code"). Please correct the citation.
  3. [Section 2.2] There are several typos and formatting glitches: "emphasizessystem- level" should be "emphasizes system-level"; "stuggle" in Section 3.2 should be "struggle"; "examing" in the Conclusion should be "examining"; and the ACM Reference Format still contains the placeholder "Make sure to enter the correct conference title from your rights confirmation email." These should be fixed before publication.
  4. [Figure 2] The workflow diagram is difficult to follow: the Yes/No branches are not labeled with the questions they answer, and the relationship between "Human Clarification," "Human Monitoring," and "Approved" is not explicit. Consider adding short branch labels or a legend to make the decision flow unambiguous.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the roadmap is a qualitative synthesis, and the only self-citation is contextual and non-load-bearing; unvalidated recommendations are evidence gaps, not circular reasoning.

full rationale

This is a qualitative vision/roadmap paper with no equations, fitted parameters, or formal derivation chain. The six-layer architecture and the four-phase workflow (Input, Orchestration, Development, Validation) are presented as an inductive synthesis of existing frameworks, supported by a broad set of external references (e.g., MetaGPT, ChatDev, CodeGen, ClarifyGPT, EvalPlus, and others). The only self-citation is [20], a prior survey by the same first author, used to note that research attention has shifted toward LLM-based agents ('Recent trends show a research shift from pure LLM models to LLM-based agents [20]') and to motivate usability challenges. This citation does not carry any of the roadmap's central claims: the framework, workflow, and recommendations do not reduce to [20]'s contents. The strongest prescriptive assertions—clarity checks, developer-in-the-loop validation, and dynamic task creation—are empirically unvalidated, and the paper itself acknowledges contrary evidence from Xu et al. [41] that developers using LLM plugins 'do not necessarily see large efficiency gains.' However, an unvalidated or even questionable empirical premise is not circularity: those claims are not defined in terms of their conclusions, and no fitted parameter or equation forces the outcome. Per the hard rules, no circular step can be exhibited with a specific reduction. Score 2 reflects a minor, non-load-bearing self-citation rather than any circular derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The paper introduces no equations or fitted parameters. Its central claim rests on a taxonomy and a proposed workflow whose benefits are asserted rather than measured. The main assumptions are domain assumptions about developer needs and unvalidated design choices, plus the self-created six-layer decomposition and the 'model cloud' and 'dynamic task creation' components.

assumptions (4)
  • domain assumption LLM-based code generation predominantly follows a 'prompt in, code out' approach.
    Used in Section 2.2 Input Phase to motivate multi-modal input and clarity checks; it is a descriptive premise about current practice.
  • domain assumption Professional programmers prioritize accuracy and robustness of generated code over raw capability.
    Introduced in Section 1 to frame why code generation quality matters; no empirical study in the paper supports the priority ordering.
  • domain assumption Frequent human interaction during development reduces black-box risk and improves explainability.
    Central design choice of the proposed workflow in Section 2.2 Development Phase; stated normatively and not tested in the paper.
  • ad hoc to paper The six-layer decomposition is a natural way to categorize existing LLM code generation works.
    The taxonomy is created for this paper without a systematic literature analysis or inter-rater validation; it is a framing assumption.
invented entities (2)
  • Model Cloud
    purpose: Stores dynamically generated agents for reuse in the orchestration phase of the proposed workflow.
    Introduced in Section 2.2 Orchestration Phase; no implementation or falsifiable prediction is provided.
  • Dynamic Task Creation
    purpose: Generates and refines tasks on demand based on execution results and human feedback during development.
    Proposed in Section 2.2 Orchestration Phase; not instantiated or evaluated in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Advancing Code Generation with Large Language Models: A Research Roadmap." pith.science (2026). https://pith.science/paper/O3VINZWZ

@misc{pith2026250111354,
  author       = {Pith},
  title        = {Pith review of: Towards Advancing Code Generation with Large Language Models: A Research Roadmap},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O3VINZWZ}},
  note         = {Machine review of arXiv:2501.11354}
}
read the original abstract

Recently, we have witnessed the rapid development of large language models, which have demonstrated excellent capabilities in the downstream task of code generation. However, despite their potential, LLM-based code generation still faces numerous technical and evaluation challenges, particularly when embedded in real-world development. In this paper, we present our vision for current research directions, and provide an in-depth analysis of existing studies on this task. We propose a six-layer vision framework that categorizes code generation process into distinct phases, namely Input Phase, Orchestration Phase, Development Phase, and Validation Phase. Additionally, we outline our vision workflow, which reflects on the currently prevalent frameworks. We systematically analyse the challenges faced by large language models, including those LLM-based agent frameworks, in code generation tasks. With these, we offer various perspectives and actionable recommendations in this area. Our aim is to provide guidelines for improving the reliability, robustness and usability of LLM-based code generation systems. Ultimately, this work seeks to address persistent challenges and to provide practical suggestions for a more pragmatic LLM-based solution for future code generation endeavors.

Figures

Figures reproduced from arXiv: 2501.11354 by the authors.

Figure 1
Figure 1. Proposed Six-Layer Architecture for LLM-Based Code Generation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our vision workflow for LLM-Based Code Generation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 2 citations worldwide. Full citation record

  1. Taxonomy of migration scenarios for Qiskit refactoring using LLMs

    cs.SE 2025-06 conditional novelty 5.0 of 10

    LLMs can generate a structured taxonomy of Qiskit migration and refactoring scenarios that largely overlaps with an expert-built taxonomy and adds some scenarios.

Reference graph

Works this paper leans on

45 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [41]

    Xu, Bogdan Vasilescu, and Graham Neubig

    Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2021. In-IDE Code Generation from Natural Language: Promise and Challenges. arXiv:2101.11149 [cs.SE] https://arxiv.org/abs/2101.11149

    arXiv 2021

  2. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [2]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    2024

  4. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  5. [4]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023)

    arXiv 2023

  6. [5]

    Zenan Chen and Jason Chan. 2024. Large language model in creative work: The role of collaboration modality and user expertise. Management Science 70, 12 (2024), 9101–9117

    work page 2024

  7. [6]

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474 (2023)

    arXiv 2023

  8. [7]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

Show all 45 references
  1. [8]

    Yucong Duan. 2024. The Large Language Model (LLM) Bias Evaluation (Age Bias).DIKWP Research Group International Standard Evaluation. DOI 10 (2024)

    2024

  2. [9]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al . 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)

    2020 arXiv

  3. [10]

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints (2023)

    2023

  4. [11]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  5. [12]

    Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. 2023. L2mac: Large language model automatic computer for unbounded code generation. arXiv preprint arXiv:2310.02003 (2023)

    2023 arXiv

  6. [13]

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352 (2023)

    2023 arXiv

  7. [14]

    Dong Huang, Qingwen Bu, and Heming Cui. 2023. Codecot and beyond: Learning to program and test like a developer. arXiv preprint arXiv:2308.08784 (2023)

    2023 arXiv

  8. [16]

    Dong Huang, Qingwen Bu, Jie Zhang, Xiaofei Xie, Junjie Chen, and Heming Cui. 2024. Bias Testing and Mitigation in LLM-based Code Generation. arXiv:2309.14345 [cs.SE] https://arxiv.org/abs/2309.14345

    2024 arXiv

  9. [17]

    Hanyao Huang, Ou Zheng, Dongdong Wang, Jiayi Yin, Zijin Wang, Shengxuan Ding, Heng Yin, Chuan Xu, Renjie Yang, Qian Zheng, et al. 2023. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. International Journal of Oral Science 15, 1 (...

    2023

  10. [18]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38

    2023

  11. [19]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515 (2024)

    2024 arXiv

  12. [20]

    Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479 (2024)

    2024 arXiv

  13. [21]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169 (2023)

    2023 arXiv

  14. [22]

    Katikapalli Subramanyam Kalyan. 2024. A survey of GPT-3 family large language models including ChatGPT and GPT-4. Natural Language Processing Journal 6 (2024), 100048

    2024

  15. [23]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  16. [24]

    Feng Lin, Dong Jae Kim, and TH Chen. 2024. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. arXiv preprint arXiv:2403.15852 (2024)

    2024 arXiv

  17. [25]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  18. [26]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)

    2021 arXiv

  19. [27]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, and Qing Wang. 2023. ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification. arXiv preprint arXiv:2310.10996 (2023)

    2023 arXiv

  20. [28]

    Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, et al. 2024. L2ceval: Evaluating language-to-code generation capabilities of large language models. Transactions of the Association for Computational Lin...

    2024

  21. [29]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  22. [30]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2024. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  23. [31]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP). 754–768. doi:10.1109/SP46214.2022.9833571

    2022

  24. [32]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatdev: Communicative agents for software development. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024

  25. [33]

    Sebastian Ruder, Anders Søgaard, and Ivan Vulić. 2019. Unsupervised cross-lingual representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts . 31–38

    2019

  26. [34]

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. arXiv:2010.03768 [cs.CL] https: //arxiv.org/abs/2010.03768

    2021 arXiv

  27. [35]

    Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) 53, 3 (2020), 1–34

    2020

  28. [36]

    Denis Weerasiri, Moshe Chai Barukh, Boualem Benatallah, Quan Z Sheng, and Rajiv Ranjan. 2017. A taxonomy and survey of cloud resource orchestration techniques. ACM Computing Surveys (CSUR) 50, 2 (2017), 1–41

    2017

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  30. [38]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer- Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)

    2023 arXiv

  31. [39]

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 (2023)

    2023 arXiv

  32. [40]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey.arXiv preprint arXiv:2309.07864 (2023)

    2023 arXiv

  33. [42]

    Zhou Yang, Zhensu Sun, Terry Zhuo Yue, Premkumar Devanbu, and David Lo. 2024. Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code. arXiv:2403.07506 [cs.SE] https://arxiv. org/abs/2403.07506

    2024 arXiv

  34. [43]

    Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. arXiv preprint arXiv:2305.04087 (2023)

    2023 arXiv

  35. [44]

    Quan Zhang, Binqi Zeng, Chijin Zhou, Gwihwan Go, Heyuan Shi, and Yu Jiang. 2024. Human-imperceptible retrieval poisoning attacks in LLM-powered applications. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering . 502–506

    2024

  36. [45]

    Simiao Zhang, Jiaping Wang, Guoliang Dong, Jun Sun, Yueling Zhang, and Geguang Pu. 2024. Experimenting a New Programming Practice with LLMs. arXiv preprint arXiv:2401.01062 (2024). , Vol. 1, No. 1, Article . Publication date: January 2025. 10 Haolin Jin, Huaming Chen, Qinghua ...

    2024 arXiv

  37. [46]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906 (2024). , Vol. 1, No. 1, Article . Publication date: January 2025

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.