{"id":"3bf6b550-2e3b-4ce0-97e8-845d1ed43ba2","arxiv_id":"2504.15080","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DLCodeGen writes deep learning project code by first predicting a structured solution plan, retrieving similar code and a code template, then using a comparison prompt to merge the best parts.","lead":"This paper presents DLCodeGen, a system that plans the structure of a deep learning project before writing the code, then blends retrieved examples with a general template. It reports about 10 percent better CodeBLEU scores than standard methods on a new benchmark, though the generated code has not been executed to confirm it works.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on CodeBLEU and a 10-case human study with no execution; Section VI.B defers pass-rate testing, so 'better code' may only mean greater similarity to reference notebooks.","rationale":"The reader's conditional verdict already centers on the absence of execution-based evaluation and the risk of data leakage from the retrieval pool. My stress-test agrees: the strongest load-bearing weakness is that all reported quantitative gains are measured without running the generated deep learning projects, and the paper explicitly defers pass-rate evaluation to future work in Section VI.B. This is not a dispute with external consensus; it is an internal missing validation step for a claim about code quality. The concrete execution test would settle whether CodeBLEU improvements translate into runnable projects, and the retrieval-pool deduplication check would settle the leakage risk. Since these are exactly the conditions under which the paper should be accepted, and the reader's CONDITIONAL verdict already encodes that requirement, I recommend no change to the verdict.","tokens_in":14143,"tokens_out":6044,"duration_ms":58388,"concrete_test":"Run an end-to-end execution test on the 100 DLCodeEval requirements: generate projects with DLCodeGen and with CEDAR/Direct at their reported temperatures, then execute each generated project in an isolated Docker container with standard TensorFlow/Keras dependencies, recording whether it imports cleanly, whether training completes at least one epoch, and whether it produces output in the requested format; compare pass rates with a paired bootstrap confidence interval. Additionally, compute the maximum BLEU/CodeBLEU overlap between each DLCodeEval reference and every DLSamplePool entry. If executable pass rates are statistically indistinguishable, or if the CodeBLEU advantage disappears after removing near-duplicate retrieval-pool entries, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DLCodeGen produces better deep learning code than direct LLM generation and RAG baselines. The evidence in Tables II and III consists entirely of CodeBLEU sub-scores and ratings from four annotators on ten manual cases; no generated project is executed. CodeBLEU rewards n-gram, AST, and dataflow overlap with a reference notebook, so a method that retrieves and copies from the same Kaggle source can inflate those scores without producing runnable code. The paper itself states in Section VI.B that test pass rate is future work, which makes the load-bearing premise—that CodeBLEU gains correspond to code that actually loads data, trains, and produces usable output—unverified. The human evaluation in Section V.B has no inter-annotator agreement statistics and covers only 10 cases, so it cannot independently carry that weight. Section V.E further tunes temperature on the test set and reports 27.96 at temperature 1.5 while Table II reports 27.52, with no run-to-run variance, weakening the precision of the claimed 9.3% improvement over CEDAR. A related leak path is also left open: the time-based split of DLCodeEval addresses LLM pretraining exposure, but DLSamplePool is drawn from the same 3,950 Notebook collection and no deduplication between DLSamplePool and DLCodeEval is reported, so retrieval-pool overlap could directly inflate CodeBLEU. If the generated code does not execute, the reported advantage does not establish the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DLCodeGen, a planning-guided pipeline for generating complete deep learning projects. A fine-tuned GPT-2 model first predicts a structured solution plan from a natural-language requirement; this plan is then used to drive two retrieval-augmented branches, one retrieving similar code samples (Code RAG) and one abstracting a code template (Template RAG); a comparative-learning prompt combines the two branches to produce the final code. The authors construct DLCodeEval from Meta Kaggle Code (100 automated test cases plus 10 manually evaluated cases), compare DLCodeGen against Direct, PS, C-CoT, and CEDAR using DeepSeek-V2.5 and GPT-4o-mini, and report consistent CodeBLEU improvements, a small human evaluation, a plan-predictor study, ablations, and a temperature sensitivity analysis. The central claimed result is that DLCodeGen outperforms state-of-the-art baselines for deep learning project generation.","tokens_in":14473,"tokens_out":4052,"duration_ms":37436,"significance":"If the headline results were robust, the paper would make a useful contribution to repository-level and deep-learning-specific code generation. The strengths are the clearly described pipeline, the public code and data repository, the broad baseline set including adapted RAG and planning methods, and the field-level evaluation of the plan predictor. However, the evaluation does not execute any generated project, and the retrieval pool and test set share a common source with no reported deduplication, so the demonstrated gains currently establish similarity to reference notebooks rather than actual runnable, correct deep learning code. The small human study and the lack of significance testing further limit the strength of the claims.","major_comments":[{"comment":"The paper explicitly states in Section VI.B that test pass rate is future work, yet the central claim that DLCodeGen generates 'better' deep learning code rests entirely on CodeBLEU sub-scores and human ratings. CodeBLEU rewards n-gram, AST, and dataflow overlap with the reference notebook, so it cannot distinguish a faithful, runnable project from one that is textually similar but non-executable. To support the central claim, the authors should add execution-based validation on at least a subset of DLCodeEval (e.g., smoke training with small data or mocked data, checking that the model compiles and runs), or alternatively narrow the claim to similarity-based code quality. This is load-bearing because every quantitative comparison in Sections V.A and V.D inherits the limitation.","section":"VI.B (Limitations) and Table II"},{"comment":"DLCodeEval and DLSamplePool are both constructed from the same 3,950 filtered TensorFlow notebooks. The time-based split addresses LLM pretraining exposure but not retrieval-pool overlap: no deduplication between DLSamplePool and DLCodeEval is reported. If a test reference notebook, or a near-duplicate of it, appears in DLSamplePool, Code RAG can retrieve it and inflate CodeBLEU relative to baselines that do not draw retrieval examples from the same source. The authors should quantify the overlap between the retrieval pool and the test set, and exclude any overlapping or near-duplicate entries before reporting the comparisons.","section":"IV.D (Datasets) and V.A (Table II)"},{"comment":"Table II reports DLCodeGen with CodeBLEU 27.52, while Figure 5 reports a peak CodeBLEU of 27.96 at temperature 1.5, and the text does not state which configuration Table II uses. Because temperature was tuned on the same DLCodeEval test set, and the main results and ablations appear to use the tuned value, the comparison is optimistically biased. The authors should report results under a predetermined configuration or through cross-validation, and provide variance estimates or significance tests for the claimed 9.3% and 7.3% improvements over CEDAR.","section":"V.E (Temperature) and Table II"},{"comment":"The human evaluation uses four annotators on only 10 cases, reports no inter-annotator agreement statistics, and describes consensus discussion after large discrepancies, which can mask genuine disagreement. Given the average advantage over CEDAR is 0.26 points on a 0–10 scale, the 3.6% improvement is fragile. The authors should report per-annotator scores, agreement metrics (e.g., Cohen's kappa or Krippendorff's alpha), and ideally a larger or preregistered evaluation before presenting the human results as supporting the central claim.","section":"V.B (Human Evaluation)"}],"minor_comments":[{"comment":"The abstract reports a 9.7% CodeBLEU improvement, while the body reports 9.3% over CEDAR for DeepSeek-V2.5 and 7.3% for GPT-4o-mini; please specify which number is aggregated and how.","section":"Abstract and Table II"},{"comment":"DLSamplePool is listed with only 40 entries, yet the text describes it as a curated collection for retrieval; clarify whether 40 is the total pool size or a representative subset, since this affects the interpretability of the Code RAG results.","section":"Table I and Section IV.D"},{"comment":"The CEDAR adaptation is limited to one retrieved code example, but the manuscript does not state how many samples Code RAG uses; the comparison should state the retrieval count for both methods to ensure fairness.","section":"Section IV.B"},{"comment":"The conclusion contains a typo: 'These plans are are utilized' should read 'These plans are utilized.'","section":"Section VII"},{"comment":"Figure 4 is referenced in Section V.C but the text of the provided manuscript does not include the figure; ensure all figures are present in the final version.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the repository availability is a plus, but the evaluation currently does not nail the central claim. I would ask the authors to either add execution-based validation, provide retrieval-pool/test-set deduplication evidence, and address the temperature and human-evaluation concerns, or substantially soften the claims. No concerns about citation practice or novelty disclosure; the related work coverage is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid systems contribution with a load-bearing evaluation gap. What is new: DLCodeGen applies a plan predictor, Code RAG, Template RAG, and a comparative-learning prompt to the task of generating entire deep-learning projects from natural language, and ships a new benchmark (DLCodeEval) plus code and data. The gains over adapted baselines are consistent across two backbone models (DeepSeek-V2.5 and GPT-4o-mini), and the ablations show the comparative mechanism earns its keep. The plan predictor result, where a fine-tuned GPT-2-large beats few-shot GPT-4o-mini on plan quality, is also a concrete plus.\n\nWhere it gets soft: the evaluation never executes any generated project. Tables II and III are CodeBLEU and a four-annotator, ten-case human study. CodeBLEU rewards n-gram and structural overlap with reference notebooks, so a retrieval-heavy method can inflate scores without producing runnable code. The paper itself concedes in Section VI.B that test pass rate is future work, which is honest but also means the headline 'better code' is really 'more similar to references.' Two additional issues compound this. First, temperature is tuned on the test set (Section V.E), so the reported optimum of 27.96 is an upper bound, not a prediction. Second, the DLSamplePool is drawn from the same 3,950-notebook collection as DLCodeEval, and the paper does not state that the pool and test set are disjoint; a retrieval-pool/test-set overlap could directly inflate CodeBLEU. The human eval has no inter-annotator agreement and too few cases to carry weight on its own.\n\nThat said, I would not call the flaws disqualifying. The method is described clearly, the comparison protocol is mostly fair (baselines adapted and re-run), and the limitations section is candid. A serious referee could push for execution-based validation and dataset-disjointness checks, and the paper would be stronger for it. The core idea is worth engaging with, and the benchmark could be useful to the community even if the reported gains shrink under stricter evaluation.\n\nRecommendation: send to peer review. It deserves a proper referee, not a desk reject. I would not cite it in the next year without seeing execution-based results, but I would bring it to a reading group to discuss evaluation methodology for generated ML code.","headline":"A plausible planning-plus-dual-RAG pipeline for whole deep-learning projects, with a new benchmark and honest limitations, but the central 'better code' claim rests on CodeBLEU and a 10-case human study, not on execution.","tokens_in":14970,"tokens_out":1414,"would_cite":false,"duration_ms":14823,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A plan-first, dual-retrieval pipeline produces deep learning project code that outperforms direct LLM generation by 9.7% on CodeBLEU.","keywords":["deep learning code generation","solution planning","retrieval-augmented generation","comparative learning","code generation benchmark","large language models","CodeBLEU"],"falsifier":"Execute the generated projects on the benchmark's test tasks and compare execution pass rates against the baselines; if DLCodeGen's pass rate is not better than direct generation, the reported gains may be n-gram and structural similarity rather than usable code. In parallel, search the retrieved code pool and the backbone model's training data for near-duplicates of the test notebooks; any substantial overlap would implicate leakage as the source of improvement.","tokens_in":13974,"feed_emoji":"🤖","tokens_out":8119,"duration_ms":67831,"temperature":0.7,"pith_summary":"The paper argues that large language models can generate complete deep learning projects, long and multi-step code that usually degrades model quality, when they are first handed a structured solution plan. The proposed system, DLCodeGen, predicts that plan with a small fine-tuned model, retrieves analogous code samples, abstracts a reusable template, and then runs a comparative step that merges both views into the final project. On a new benchmark built from real deep learning notebooks, the paper reports a 9.7% higher CodeBLEU score than the best baseline and better human ratings for requirement compliance and idiomaticity. If the claim holds, it would let developers produce coherent, domain-aware AI code from a single natural-language request.","feed_headline":"A plan-first pipeline lifts CodeBLEU by 9.7% for deep-learning code","feed_subtitle":"A solution plan plus two retrieval routes helps LLMs write longer, domain-specific AI projects.","key_machinery":"The central mechanism is solution-plan prediction: a comparatively small language model, fine-tuned on a corpus of requirement-plan pairs, turns a natural-language request into a structured plan with fields for task category, dataset attributes, preprocessing, model architecture, and hyperparameters. The plan does three jobs at once: it selects the correct task-specific subset of the code pool, it acts as the query for BM25 similarity retrieval of concrete code samples, and it anchors the final comparison prompt. The load-bearing integration step is comparative generation: given the plan and two drafts, one from Code RAG and one from Template RAG, the LLM is prompted to select the best components of each and synthesize the final project, which the ablation study identifies as the largest single contributor to the gain.","core_discovery":"On its own terms, the paper claims that the bottleneck in deep learning project generation is not the generative capacity of the LLM but the absence of a global blueprint. The blueprint is a solution plan covering task category, dataset shape, preprocessing, model architecture, and hyperparameters; it supplies the contextual coherence that open-domain LLMs lack. The paper further claims that two complementary retrieval views are necessary, concrete similar code for implementation detail and an abstracted template for structure, and that a comparative learning mechanism, which asks the LLM to choose the best segments of each against the plan, is what makes the combination succeed. The evidence is a consistent CodeBLEU improvement over planning, chain-of-thought, and retrieval baselines on two different backbone LLMs.","pith_inferences":["Because the paper's benchmark does not execute generated code, the natural next test is pass-rate evaluation; the authors explicitly postpone that to future work.","The comparative prompt could be reused beyond its two RAG drafts, for example to merge outputs from different models, temperatures, or programming styles into one project.","The DLCodeEval construction, time-filtered and leakage-checked, offers a reusable template for building domain code benchmarks, though its test set of 100 tasks is small enough that sampling error should be kept in mind."],"forward_implications":["A structured plan improves generation of long, multi-stage code, so the same recipe should transfer to other step-heavy code domains such as data pipelines.","Two-level retrieval, concrete samples plus abstracted templates, outperforms either alone, suggesting both precision and generalization are needed.","A small domain-tuned planner can beat much larger general LLMs at producing task plans, making plan prediction a low-cost improvement lever.","The comparative step is the largest contributor to the gain; removing it costs more than removing either retrieval route.","With a plan and comparison guardrails, a higher sampling temperature (1.5) becomes optimal for code, contrary to the usual low-temperature default."],"supporting_citations":[{"why":"Supplies the chain-of-thought motivation: intermediate reasoning plans improve LLM task comprehension.","marker":"[13]"},{"why":"Provides the retrieval-augmented generation basis that the two RAG strategies extend.","marker":"[17]"},{"why":"Plan-and-Solve prompting is both a baseline and the planning paradigm DLCodeGen refines with a trained predictor.","marker":"[27]"},{"why":"CEDAR is the strongest retrieval-based baseline; DLCodeGen is compared directly against it.","marker":"[34]"},{"why":"BM25 is the retrieval scoring algorithm used to rank similar code samples.","marker":"[36]"},{"why":"CodeBLEU is the automatic metric used for the headline improvements.","marker":"[38]"},{"why":"The public notebook collection from which the plan corpus and evaluation benchmark are built.","marker":"[43]"},{"why":"Automatic chain-of-thought prompting is the C-CoT baseline used in comparisons.","marker":"[15]"}],"fun_headline_variants":["Plan-guided recipe lifts deep-learning CodeBLEU by 9.7%","Blueprint plus retrieval sharpens LLM-written deep-learning code","DLCodeGen: planning ahead yields 9.7% better deep-learning projects","Two retrieval routes and one plan: how to boost AI code quality","Planning-first method beats baselines on deep-learning code generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that CodeBLEU and human ratings measure code quality well enough without running the generated code, and that the time-based split keeps test projects out of both the retrieval pool and the base model's training data.","fun_headline_variants_meta":{"raw":{"variants":["Plan-guided recipe lifts deep-learning CodeBLEU by 9.7%","Blueprint plus retrieval sharpens LLM-written deep-learning code","DLCodeGen: planning ahead yields 9.7% better deep-learning projects","Two retrieval routes and one plan: how to boost AI code quality","Planning-first method beats baselines on deep-learning code generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000964,"raw_usage":{"total_tokens":4074,"prompt_tokens":887,"completion_tokens":3187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":3095}},"tokens_in":503,"tokens_out":3187,"duration_ms":21019,"temperature":1.0,"reasoning_tokens":3095,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:33:28.064406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Execute the generated projects on the benchmark's test tasks and compare execution pass rates against the baselines; if DLCodeGen's pass rate is not better than direct generation, the reported gains may be n-gram and structural similarity rather than usable code. In parallel, search the retrieved code pool and the backbone model's training data for near-duplicates of the test notebooks; any substantial overlap would implicate leakage as the source of improvement.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive NLP tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the retrieval-augmented generation basis that the two RAG strategies extend."},{"cited_title":"Plan- and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,","cited_arxiv_id":null,"evidence_quote":"Plan-and-Solve prompting is both a baseline and the planning paradigm DLCodeGen refines with a trained predictor."},{"cited_title":"Retrieval-based prompt selec- tion for code-related few-shot learning,","cited_arxiv_id":null,"evidence_quote":"CEDAR is the strongest retrieval-based baseline; DLCodeGen is compared directly against it."}],"review_version":1}