Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs

Beijun Shen; Chen Xie; Mingsheng Jiao; Xiaodong Gu

REVIEW 4 major objections 5 minor 43 references

Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A plan-first, dual-retrieval pipeline produces deep learning project code that outperforms direct LLM generation by 9.7% on CodeBLEU.

desk verdict A plausible planning-plus-dual-RAG pipeline for whole deep-learning projects, with a new benchmark and honest limitations, but the central 'better code' claim rests on CodeBLEU and a 10-case human study, not on execution. read the letter →

arxiv 2504.15080 v1 pith:LAYSNEYI submitted 2025-04-21 cs.SE cs.AI

Chen Xie , Mingsheng Jiao , Xiaodong Gu , Beijun Shen This is my paper · ORCID

classification cs.SEcs.AI

keywords deeplearningcodegenerationsolutionplanningretrieval-augmentedcomparativebenchmarklargelanguagemodelsBLEU

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large language models can generate complete deep learning projects, long and multi-step code that usually degrades model quality, when they are first handed a structured solution plan. The proposed system, DLCodeGen, predicts that plan with a small fine-tuned model, retrieves analogous code samples, abstracts a reusable template, and then runs a comparative step that merges both views into the final project. On a new benchmark built from real deep learning notebooks, the paper reports a 9.7% higher CodeBLEU score than the best baseline and better human ratings for requirement compliance and idiomaticity. If the claim holds, it would let developers produce coherent, domain-aware AI code from a single natural-language request.

What carries the argument

The central mechanism is solution-plan prediction: a comparatively small language model, fine-tuned on a corpus of requirement-plan pairs, turns a natural-language request into a structured plan with fields for task category, dataset attributes, preprocessing, model architecture, and hyperparameters. The plan does three jobs at once: it selects the correct task-specific subset of the code pool, it acts as the query for BM25 similarity retrieval of concrete code samples, and it anchors the final comparison prompt. The load-bearing integration step is comparative generation: given the plan and two drafts, one from Code RAG and one from Template RAG, the LLM is prompted to select the best components of each and synthesize the final project, which the ablation study identifies as the largest single contributor to the gain.

What would settle it

Execute the generated projects on the benchmark's test tasks and compare execution pass rates against the baselines; if DLCodeGen's pass rate is not better than direct generation, the reported gains may be n-gram and structural similarity rather than usable code. In parallel, search the retrieved code pool and the backbone model's training data for near-duplicates of the test notebooks; any substantial overlap would implicate leakage as the source of improvement.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the bottleneck in deep learning project generation is not the generative capacity of the LLM but the absence of a global blueprint. The blueprint is a solution plan covering task category, dataset shape, preprocessing, model architecture, and hyperparameters; it supplies the contextual coherence that open-domain LLMs lack. The paper further claims that two complementary retrieval views are necessary, concrete similar code for implementation detail and an abstracted template for structure, and that a comparative learning mechanism, which asks the LLM to choose the best segments of each against the plan, is what makes the combination succeed. The evidence is a consistent CodeBLEU improvement over planning, chain-of-thought, and retrieval baselines on two different backbone LLMs.

Load-bearing premise

The approach assumes that CodeBLEU and human ratings measure code quality well enough without running the generated code, and that the time-based split keeps test projects out of both the retrieval pool and the base model's training data.

Editorial extensions

If this is right

A structured plan improves generation of long, multi-stage code, so the same recipe should transfer to other step-heavy code domains such as data pipelines.
Two-level retrieval, concrete samples plus abstracted templates, outperforms either alone, suggesting both precision and generalization are needed.
A small domain-tuned planner can beat much larger general LLMs at producing task plans, making plan prediction a low-cost improvement lever.
The comparative step is the largest contributor to the gain; removing it costs more than removing either retrieval route.
With a plan and comparison guardrails, a higher sampling temperature (1.5) becomes optimal for code, contrary to the usual low-temperature default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the paper's benchmark does not execute generated code, the natural next test is pass-rate evaluation; the authors explicitly postpone that to future work.
The comparative prompt could be reused beyond its two RAG drafts, for example to merge outputs from different models, temperatures, or programming styles into one project.
The DLCodeEval construction, time-filtered and leakage-checked, offers a reusable template for building domain code benchmarks, though its test set of 100 tasks is small enough that sampling error should be kept in mind.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a solid systems contribution with a load-bearing evaluation gap. What is new: DLCodeGen applies a plan predictor, Code RAG, Template RAG, and a comparative-learning prompt to the task of generating entire deep-learning projects from natural language, and ships a new benchmark (DLCodeEval) plus code and data. The gains over adapted baselines are consistent across two backbone models (DeepSeek-V2.5 and GPT-4o-mini), and the ablations show the comparative mechanism earns its keep. The plan predictor result, where a fine-tuned GPT-2-large beats few-shot GPT-4o-mini on plan quality, is also a concrete plus.

Where it gets soft: the evaluation never executes any generated project. Tables II and III are CodeBLEU and a four-annotator, ten-case human study. CodeBLEU rewards n-gram and structural overlap with reference notebooks, so a retrieval-heavy method can inflate scores without producing runnable code. The paper itself concedes in Section VI.B that test pass rate is future work, which is honest but also means the headline 'better code' is really 'more similar to references.' Two additional issues compound this. First, temperature is tuned on the test set (Section V.E), so the reported optimum of 27.96 is an upper bound, not a prediction. Second, the DLSamplePool is drawn from the same 3,950-notebook collection as DLCodeEval, and the paper does not state that the pool and test set are disjoint; a retrieval-pool/test-set overlap could directly inflate CodeBLEU. The human eval has no inter-annotator agreement and too few cases to carry weight on its own.

That said, I would not call the flaws disqualifying. The method is described clearly, the comparison protocol is mostly fair (baselines adapted and re-run), and the limitations section is candid. A serious referee could push for execution-based validation and dataset-disjointness checks, and the paper would be stronger for it. The core idea is worth engaging with, and the benchmark could be useful to the community even if the reported gains shrink under stricter evaluation.

Recommendation: send to peer review. It deserves a proper referee, not a desk reject. I would not cite it in the next year without seeing execution-based results, but I would bring it to a reading group to discuss evaluation methodology for generated ML code.

Referee Report

4 major / 5 minor

Summary. The paper proposes DLCodeGen, a planning-guided pipeline for generating complete deep learning projects. A fine-tuned GPT-2 model first predicts a structured solution plan from a natural-language requirement; this plan is then used to drive two retrieval-augmented branches, one retrieving similar code samples (Code RAG) and one abstracting a code template (Template RAG); a comparative-learning prompt combines the two branches to produce the final code. The authors construct DLCodeEval from Meta Kaggle Code (100 automated test cases plus 10 manually evaluated cases), compare DLCodeGen against Direct, PS, C-CoT, and CEDAR using DeepSeek-V2.5 and GPT-4o-mini, and report consistent CodeBLEU improvements, a small human evaluation, a plan-predictor study, ablations, and a temperature sensitivity analysis. The central claimed result is that DLCodeGen outperforms state-of-the-art baselines for deep learning project generation.

Significance. If the headline results were robust, the paper would make a useful contribution to repository-level and deep-learning-specific code generation. The strengths are the clearly described pipeline, the public code and data repository, the broad baseline set including adapted RAG and planning methods, and the field-level evaluation of the plan predictor. However, the evaluation does not execute any generated project, and the retrieval pool and test set share a common source with no reported deduplication, so the demonstrated gains currently establish similarity to reference notebooks rather than actual runnable, correct deep learning code. The small human study and the lack of significance testing further limit the strength of the claims.

major comments (4)

[VI.B (Limitations) and Table II] The paper explicitly states in Section VI.B that test pass rate is future work, yet the central claim that DLCodeGen generates 'better' deep learning code rests entirely on CodeBLEU sub-scores and human ratings. CodeBLEU rewards n-gram, AST, and dataflow overlap with the reference notebook, so it cannot distinguish a faithful, runnable project from one that is textually similar but non-executable. To support the central claim, the authors should add execution-based validation on at least a subset of DLCodeEval (e.g., smoke training with small data or mocked data, checking that the model compiles and runs), or alternatively narrow the claim to similarity-based code quality. This is load-bearing because every quantitative comparison in Sections V.A and V.D inherits the limitation.
[IV.D (Datasets) and V.A (Table II)] DLCodeEval and DLSamplePool are both constructed from the same 3,950 filtered TensorFlow notebooks. The time-based split addresses LLM pretraining exposure but not retrieval-pool overlap: no deduplication between DLSamplePool and DLCodeEval is reported. If a test reference notebook, or a near-duplicate of it, appears in DLSamplePool, Code RAG can retrieve it and inflate CodeBLEU relative to baselines that do not draw retrieval examples from the same source. The authors should quantify the overlap between the retrieval pool and the test set, and exclude any overlapping or near-duplicate entries before reporting the comparisons.
[V.E (Temperature) and Table II] Table II reports DLCodeGen with CodeBLEU 27.52, while Figure 5 reports a peak CodeBLEU of 27.96 at temperature 1.5, and the text does not state which configuration Table II uses. Because temperature was tuned on the same DLCodeEval test set, and the main results and ablations appear to use the tuned value, the comparison is optimistically biased. The authors should report results under a predetermined configuration or through cross-validation, and provide variance estimates or significance tests for the claimed 9.3% and 7.3% improvements over CEDAR.
[V.B (Human Evaluation)] The human evaluation uses four annotators on only 10 cases, reports no inter-annotator agreement statistics, and describes consensus discussion after large discrepancies, which can mask genuine disagreement. Given the average advantage over CEDAR is 0.26 points on a 0–10 scale, the 3.6% improvement is fragile. The authors should report per-annotator scores, agreement metrics (e.g., Cohen's kappa or Krippendorff's alpha), and ideally a larger or preregistered evaluation before presenting the human results as supporting the central claim.

minor comments (5)

[Abstract and Table II] The abstract reports a 9.7% CodeBLEU improvement, while the body reports 9.3% over CEDAR for DeepSeek-V2.5 and 7.3% for GPT-4o-mini; please specify which number is aggregated and how.
[Table I and Section IV.D] DLSamplePool is listed with only 40 entries, yet the text describes it as a curated collection for retrieval; clarify whether 40 is the total pool size or a representative subset, since this affects the interpretability of the Code RAG results.
[Section IV.B] The CEDAR adaptation is limited to one retrieved code example, but the manuscript does not state how many samples Code RAG uses; the comparison should state the retrieval count for both methods to ensure fairness.
[Section VII] The conclusion contains a typo: 'These plans are are utilized' should read 'These plans are utilized.'
[Figure 4] Figure 4 is referenced in Section V.C but the text of the provided manuscript does not include the figure; ensure all figures are present in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims are empirical benchmark comparisons, not derivations from their own inputs.

full rationale

The paper does not present a formal derivation chain; its central claim is an empirical comparison on a constructed benchmark. DLCodeGen's pipeline (GPT-2 plan predictor trained on DLPlanData, BM25 retrieval from DLSamplePool, template abstraction, and comparative prompting) is evaluated against baselines on the same DLCodeEval set using CodeBLEU and human ratings. None of the reported equations or metrics reduce by construction to a fitted parameter: CodeBLEU compares generated code to reference code, and the method's components are studied through ablation rather than assumed. The self-citations present (e.g., [40] for EM and [11] for code-generation background) are definitional or background citations and are not load-bearing. Section VI.B's acknowledged residual data-leakage risk and deferral of pass-rate testing are genuine threats to external validity, but they do not make the comparison definitionally circular, because the reported gains are not entailed by the construction of the benchmark or by the cited prior work. Therefore no significant circularity is found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are postulated; the solution plan, template, and comparison prompt are software artifacts, not theoretical entities. Free parameters are limited to decoding and temperature choices, with the temperature selected on the test set being the most consequential.

free parameters (4)

temperature = 1.5
Selected by maximizing CodeBLEU on the DLCodeEval test set in RQ5 (Figure 5), after trying 0.0, 0.5, 1.0, 1.5, 2.0.
num_beams = 3
Chosen for the GPT-2 plan predictor's beam search decoding; no sensitivity analysis is reported.
retrieved_code_count = top two for template; k unspecified for code RAG
Template RAG uses the top two retrieved samples; Code RAG's k is not precisely specified in the text.
plan_predictor_hyperparameters = lr=1e-3, batch=16, epochs=8
Standard tuning choices, reported in Section IV.E; not shown to be optimal.

assumptions (4)

domain assumption The four-component solution plan schema (task category, dataset, preprocess, model architecture) is sufficient to guide generation of all deep learning projects.
Section III.B defines the plan structure and uses it as the global blueprint for all tasks in the benchmark.
domain assumption LLM-extracted user requirements and solution plans from Kaggle notebooks are accurate enough to train a predictor and build a benchmark.
Section IV.D describes using an LLM to extract and summarize requirements and plans; this introduces an unverified dependency on LLM output quality.
domain assumption BM25 lexical similarity between solution plans identifies semantically relevant code samples.
Section III.C defines two-stage retrieval using task category narrowing and BM25; no comparison with semantic embedding retrieval is provided.
domain assumption Post-2024 timestamps and LLM-based verification ensure the test notebooks were not seen during LLM training.
Section IV.D and VI.B claim this reduces data leakage; the limitation section admits residual risk.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs." pith.science (2026). https://pith.science/paper/LAYSNEYI

@misc{pith2026250415080,
  author       = {Pith},
  title        = {Pith review of: Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LAYSNEYI}},
  note         = {Machine review of arXiv:2504.15080}
}

read the original abstract

While large language models (LLMs) have been widely applied to code generation, they struggle with generating entire deep learning projects, which are characterized by complex structures, longer functions, and stronger reliance on domain knowledge than general-purpose code. An open-domain LLM often lacks coherent contextual guidance and domain expertise for specific projects, making it challenging to produce complete code that fully meets user requirements. In this paper, we propose a novel planning-guided code generation method, DLCodeGen, tailored for generating deep learning projects. DLCodeGen predicts a structured solution plan, offering global guidance for LLMs to generate the project. The generated plan is then leveraged to retrieve semantically analogous code samples and subsequently abstract a code template. To effectively integrate these multiple retrieval-augmented techniques, a comparative learning mechanism is designed to generate the final code. We validate the effectiveness of our approach on a dataset we build for deep learning code generation. Experimental results demonstrate that DLCodeGen outperforms other baselines, achieving improvements of 9.7% in CodeBLEU and 3.6% in human evaluation metrics.

Figures

Figures reproduced from arXiv: 2504.15080 by the authors.

**Figure 1.** Overview of DLCodeGen of the planning process, ensuring a more robust alignment with the complexities of deep learning tasks. C. Retrieval-Augmented Generation Retrieval-augmented generation (RAG) [17] improves output quality by retrieving relevant information from external sources and integrating it with the original query. This approach is particularly effective for knowledge-intensive tasks like code generation… view at source ↗

**Figure 3.** An Illustration of Comparative Generation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 2.** An example of a solution plan for a deep learning project 4) Model Architecture: Specifies the structural configuration of the model, including layer compositions, connectivity patterns, and the initialization and optimization of hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance of Solution Plan Predictor fine-tuning, emphasizing the necessity of tailored optimization for domain-specific applications to achieve optimal results. To further examine the performance differences among various methods in predicting deep learning plans, w…

**Figure 5.** Figure 5: The Trend of Scores with Temperature Changes [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 28 canonical work pages

[1]

Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,

T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,” ACM Comput. Surv., vol. 52, no. 4, pp. 65:1–65:43, 2019

work page 2019
[2]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017, pp. 5998–6008

2017
[3]

Deep learning for computer vision: A brief review,

A. V oulodimos, N. Doulamis, A. D. Doulamis, and E. Protopapadakis, “Deep learning for computer vision: A brief review,” Comput. Intell. Neurosci., pp. 7 068 349:1–7 068 349:13, 2018

work page 2018
[4]

Recent trends in deep learning based natural language processing,

T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent trends in deep learning based natural language processing,” IEEE Comput. Intell. Mag., vol. 13, no. 3, pp. 55–75, 2018

work page 2018
[5]

Large language models for software engineer- ing: A systematic literature review,

X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineer- ing: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 220:1–220:79, 2024

work page 2024
[6]

Competition-level code generation with alphacode,

Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling et al. , “Competition-level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022

work page 2022
[7]

Multi-task learning based pre-trained language model for code completion,

F. Liu, G. Li, Y . Zhao, and Z. Jin, “Multi-task learning based pre-trained language model for code completion,” in ASE, 2020, pp. 473–485

work page 2020
[8]

Intellicode compose: code generation using transformer,

A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: code generation using transformer,” in ESEC/FSE, 2020, pp. 1433–1443

work page 2020

Show all 43 references

[9]

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” in NeurIPS, 2023

2023
[10]

Self-collaboration code generation via chatgpt,

Y . Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via chatgpt,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 7, 2024

2024
[11]

Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in ICSE, 2024, pp. 37:1– 37:12

2024
[12]

Evaluating chatgpt-3.5 efficiency in solving coding problems of different complexity levels: An empirical analysis,

M. Li and B. Krishnamachari, “Evaluating chatgpt-3.5 efficiency in solving coding problems of different complexity levels: An empirical analysis,” CoRR, vol. abs/2411.07529, 2024

2024 arXiv
[13]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in NeurIPS, 2022

2022
[14]

Structured chain-of-thought prompting for code generation,

J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” ACM Trans. Softw. Eng. Methodol. , 2024

2024
[15]

Automatic chain of thought prompting in large language models,

Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” in ICLR, 2023

2023
[16]

Self-planning code generation with large language models,

X. Jiang, Y . Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 7, pp. 1–30, 2024

2024
[17]

Retrieval-augmented generation for knowledge-intensive NLP tasks,

P. S. H. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in NeurIPS, 2020

2020
[18]

The WEKA data mining software: an update,

M. A. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The WEKA data mining software: an update,” SIGKDD Explor., vol. 11, no. 1, pp. 10–18, 2009

2009
[19]

Dash: A low code development platform for AI applications in industry,

Y . Wang, W. Song, Y . Yang, C. Mahmoudi, S. Shekhar, and K. P. Birman, “Dash: A low code development platform for AI applications in industry,” in UEMCON, 2023, pp. 72–81

2023
[20]

The PHOTON wizard - towards educa- tional machine learning code generators,

R. Leenings, N. R. Winter, K. Sarink, J. Ernsting, X. Jiang, U. Dannlowski, and T. Hahn, “The PHOTON wizard - towards educa- tional machine learning code generators,” CoRR, vol. abs/2002.05432, 2020

2002 arXiv
[21]

AI for low-code for AI,

N. Rao, J. Tsay, K. Kate, V . J. Hellendoorn, and M. Hirzel, “AI for low-code for AI,” in IUI, 2024, pp. 837–852

2024
[22]

Large language models synergize with automated machine learning,

J. Xu, J. Li, Z. Liu, N. A. V . Suryanarayanan, G. Zhou, J. Guo, H. Iba, and K. Tei, “Large language models synergize with automated machine learning,” CoRR, vol. abs/2405.03727, 2024

2024 arXiv
[23]

Large language models for constructing and optimizing machine learning workflows: A survey,

Y . Gu, H. You, J. Cao, and M. Yu, “Large language models for constructing and optimizing machine learning workflows: A survey,” CoRR, vol. abs/2411.10478, 2024

2024 arXiv
[24]

Automl-gpt: Automatic machine learning with GPT,

S. Zhang, C. Gong, L. Wu, X. Liu, and M. Zhou, “Automl-gpt: Automatic machine learning with GPT,” CoRR, vol. abs/2305.02499, 2023

2023 arXiv
[25]

Mlcopilot: Unleashing the power of large language models in solving machine learning tasks,

L. Zhang, Y . Zhang, K. Ren, D. Li, and Y . Yang, “Mlcopilot: Unleashing the power of large language models in solving machine learning tasks,” in EACL, 2024, pp. 2931–2959

2024
[26]

Hugginggpt: Solving AI tasks with chatgpt and its friends in hugging face,

Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “Hugginggpt: Solving AI tasks with chatgpt and its friends in hugging face,” in NeurIPS, 2023

2023
[27]

Plan- and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,

L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K. Lee, and E. Lim, “Plan- and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,” in ACL, 2023, pp. 2609–2634

2023
[28]

Debug like a human: A large language model debugger via verifying runtime execution step by step,

L. Zhong, Z. Wang, and J. Shang, “Debug like a human: A large language model debugger via verifying runtime execution step by step,” in Findings of ACL , 2024, pp. 851–870

2024
[29]

Selfevolve: A code evolution frame- work via large language models,

S. Jiang, Y . Wang, and Y . Wang, “Selfevolve: A code evolution frame- work via large language models,” CoRR, vol. abs/2306.02907, 2023

2023 arXiv
[30]

Siren’s song in the AI ocean: A survey on hallucination in large language models,

Y . Zhang, Y . Li, L. Cui, D. Cai, L. Liu, T. Fu et al., “Siren’s song in the AI ocean: A survey on hallucination in large language models,” CoRR, vol. abs/2309.01219, 2023

2023 arXiv
[31]

Towards continual knowledge learning of language models,

J. Jang, S. Ye, S. Yang, J. Shin, J. Han, G. Kim, S. J. Choi, and M. Seo, “Towards continual knowledge learning of language models,” in ICLR, 2022

2022
[32]

Retrieval augmented code generation and summarization,

M. R. Parvez, W. U. Ahmad, S. Chakraborty, B. Ray, and K. Chang, “Retrieval augmented code generation and summarization,” in Findings of EMNLP, 2021, pp. 2719–2734

2021
[33]

Reacc: A retrieval-augmented code completion framework,

S. Lu, N. Duan, H. Han, D. Guo, S. Hwang, and A. Svyatkovskiy, “Reacc: A retrieval-augmented code completion framework,” in ACL, 2022, pp. 6227–6240

2022
[34]

Retrieval-based prompt selec- tion for code-related few-shot learning,

N. Nashid, M. Sintaha, and A. Mesbah, “Retrieval-based prompt selec- tion for code-related few-shot learning,” in ICSE, 2023, pp. 2450–2462

2023
[35]

Docprompting: Generating code by retrieving the docs,

S. Zhou, U. Alon, F. F. Xu, Z. Jiang, and G. Neubig, “Docprompting: Generating code by retrieving the docs,” in ICLR, 2023

2023
[36]

Okapi at trec-3,

S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gat- ford et al., “Okapi at trec-3,” Nist Special Publication Sp , vol. 109, p. 109, 1995

1995
[37]

Learning by thinking in natural and artificial minds,

T. Lombrozo, “Learning by thinking in natural and artificial minds,” Trends in Cognitive Sciences , vol. 28, p. 1, 2024

2024
[38]

Codebleu: a method for automatic evaluation of code synthesis,

S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” CoRR, vol. abs/2009.10297, 2020

2009 arXiv
[39]

Bleu: a method for automatic evaluation of machine translation,

K. Papineni, S. Roukos, T. Ward, and W. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in ACL, 2002, pp. 311– 318

2002
[40]

Infere: Step-by-step regex generation via chain of inference,

S. Zhang, X. Gu, Y . Chen, and B. Shen, “Infere: Step-by-step regex generation via chain of inference,” in ASE. IEEE, 2023, pp. 1505– 1515

2023
[41]

Neu- ral machine translation for mathematical formulae,

F. Petersen, M. Schubotz, A. Greiner-Petter, and B. Gipp, “Neu- ral machine translation for mathematical formulae,” arXiv preprint arXiv:2305.16433, 2023

2023 arXiv
[42]

Informer: Beyond efficient transformer for long sequence time-series forecasting,

H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in AAAI, vol. 35, no. 12, 2021, pp. 11 106–11 115

2021
[43]

Meta kaggle code,

J. Plotts and M. Risdal, “Meta kaggle code,” 2023. [Online]. Available: https://www.kaggle.com/ds/3240808

2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,

T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,” ACM Comput. Surv., vol. 52, no. 4, pp. 65:1–65:43, 2019

work page 2019

[2] [2]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017, pp. 5998–6008

2017

[3] [3]

Deep learning for computer vision: A brief review,

A. V oulodimos, N. Doulamis, A. D. Doulamis, and E. Protopapadakis, “Deep learning for computer vision: A brief review,” Comput. Intell. Neurosci., pp. 7 068 349:1–7 068 349:13, 2018

work page 2018

[4] [4]

Recent trends in deep learning based natural language processing,

T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent trends in deep learning based natural language processing,” IEEE Comput. Intell. Mag., vol. 13, no. 3, pp. 55–75, 2018

work page 2018

[5] [5]

Large language models for software engineer- ing: A systematic literature review,

X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineer- ing: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 220:1–220:79, 2024

work page 2024

[6] [6]

Competition-level code generation with alphacode,

Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling et al. , “Competition-level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022

work page 2022

[7] [7]

Multi-task learning based pre-trained language model for code completion,

F. Liu, G. Li, Y . Zhao, and Z. Jin, “Multi-task learning based pre-trained language model for code completion,” in ASE, 2020, pp. 473–485

work page 2020

[8] [8]

Intellicode compose: code generation using transformer,

A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: code generation using transformer,” in ESEC/FSE, 2020, pp. 1433–1443

work page 2020