{"id":"986792e0-dffd-4c9b-a5e1-12e0d7f4c764","arxiv_id":"2507.12308","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Chain-of-Descriptions, a plan-then-generate prompt strategy, improves VHDL code generation and summarization scores for eight LLMs on the VHDL-Eval and VHDL-Xform benchmarks.","lead":"The paper tests eight large language models on VHDL code generation and summarization, finds all of them underperform, and proposes Chain-of-Descriptions, a prompting strategy that asks the model to write a step-by-step plan before producing the final code or summary. The strategy improves scores on two benchmarks, but the comparison mixes in extra sampling and different decoding settings, so the headline gain is not cleanly attributed to the method.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoDes gains may be a sampling-budget artifact: plan formulation uses T=0.89, top-k=50, and up to 3 retries versus baseline single sample at T=0.5/0.7, top-k=40 (App. D, E.1), so the reported Pass@1 lifts do not isolate the descriptive plan.","rationale":"I focused on the attribution of CoDes' improvement because the paper's title and central claim concern improving LLMs for VHDL code generation and summarization. The strongest evidence is the consistent lifts across eight models in Tables 1–2, and the SEC check is rigorous. However, the experimental design never compares CoDes with a baseline matched in sampling budget, decoding hyperparameters, and number of LLM calls. Appendix D reveals the hyperparameter split; Appendix E.1/H reveal the retry mechanism; Section 5.3.2 shows Multi-Step uses separate prompts. Any of these differences can change Pass@1 substantially, especially at low baseline rates. The paper even states zero-shot uses single sampling while plan formulation may sample up to three times, so the comparison is not apples-to-apples. This is an internal control problem, not a disagreement with external consensus. The proposed concrete test is minimal and would settle the attribution: force CoDes to use the baseline's decoding settings and a single plan attempt. If the lift survives, the plan structure is doing real work; if not, the claim should be weakened to 'CoDes with additional sampling and tuned decoding improves performance,' which is a less novel result. I therefore do not move the reader's verdict—the paper remains a conditional accept pending this control, since the issue is addressable and the SEC-based verification is a genuine strength. The reader identified the same underlying confound, so my assessment agrees.","tokens_in":14214,"tokens_out":4486,"duration_ms":49711,"concrete_test":"On VHDL-Eval, re-run the Granite-Code-34b CoDes pipeline with plan formulation forced to the zero-shot decoding settings (T=0.5, top-k=40), a single plan sample, no retry, and the same one-sample final execution; compare Pass@1 (SEC) against the Table 1 baseline of 0.187. If the matched CoDes Pass@1 does not exceed the baseline by more than sampling noise, the reported 0.246 gain is attributable to sampling budget and hyperparameters, not to the chain of descriptions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that CoDes 'significantly surpasses' standard prompting—rests on Tables 1 vs 2, e.g., Granite-Code-34b Pass@1 (TB) rising from 0.192 to 0.254 and Pass@1 (SEC) from 0.187 to 0.246. But the CoDes condition is not matched to the baseline in sampling behavior. Appendix D states zero-shot results use a single sample at T=0.5 or 0.7 with top-k=40, while the plan formulation phase uses T=0.89 and top-k=50. Appendix E.1 and H further reveal that if the generated plan does not match the expected extraction pattern, sampling is repeated up to three times. In addition, Table 2 uses Multi-Step execution, which issues at least two separate LLM calls (plan and execute), versus one call for the zero-shot baseline. The observed improvement could therefore be driven by (i) a more favorable temperature/top-k, (ii) the retry-until-parse mechanism, or (iii) the extra inference compute of a second LLM call—none of which tests the value of the intermediate descriptive steps themselves. The paper also reports no confidence intervals or significance tests, so 'significantly surpasses' is not statistically supported. This confound is load-bearing because the paper's central contribution is the CoDes method itself: if the gain vanishes under matched sampling, the main claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates eight code, instruct, and chat LLMs on two VHDL benchmarks: VHDL-Eval, a 202-problem benchmark with self-verifying testbenches and reference solutions, and VHDL-Xform, a newly introduced in-house dataset of 6,500 code-clone pairs used to probe functional-equivalence understanding. The evaluation reports Pass@1 using both testbenches and sequential equivalence checking (SEC), a self-consistency score, ROUGE-L, and an LLM judge-based preference rate. After documenting consistent underperformance of existing LLMs, the authors propose Chain-of-Descriptions (CoDes), which first prompts the LLM to generate intermediate descriptive steps for a problem statement or code snippet, post-processes these steps into a plan, and then feeds the refined plan together with the original input to the LLM to produce the final code or summary. Table 2 reports consistent improvements for CoDes over the zero-shot baseline, and Section 6 presents ablations on prompt length, AST-based versus line-based planning, and single-step versus multi-step execution.","tokens_in":14539,"tokens_out":7050,"duration_ms":80218,"significance":"The paper makes useful contributions to an under-studied area: it introduces VHDL-Xform, evaluates a broader set of LLMs than most prior HDL work, and uses both self-verifying testbenches and formal SEC to assess functional correctness rather than relying on compile-only checks. The self-consistency metric and the judge-LLM preference rate are also reasonable evaluation ideas. However, the central quantitative claim that CoDes 'significantly surpasses' standard prompting is not yet established, because the CoDes condition is not matched to the baseline in sampling budget, decoding hyperparameters, retry behavior, or number of LLM calls. If a matched-control experiment confirms that the descriptive plan itself drives the gains, this framework would be a practical and reproducible contribution to LLM-assisted VHDL design; as it stands, the main comparison is confounded.","major_comments":[{"comment":"The headline comparison between zero-shot and CoDes is not matched in sampling behavior. Appendix D states that zero-shot results use a single sample at T=0.5 or 0.7 with top-k=40, whereas the CoDes plan formulation uses T=0.89, top-k=50, and up to three sampling attempts before accepting a parseable plan (Appendices E.1 and H), and Multi-Step execution adds a second LLM call (Section 5.3.2). Consequently the reported Pass@1 improvements (for example, Granite-Code-34b from 0.192 to 0.254 on testbenches and from 0.187 to 0.246 on SEC) may be caused by the larger sampling budget, the different temperature, the retry-until-extract mechanism, or the extra inference compute, rather than by the descriptive plan itself. This concern is load-bearing because the paper's central claim is that the CoDes plan structure improves performance. Please add a control in which the baseline receives exactly the same decoding hyperparameters, the same number of sampling attempts, and the same number of LLM calls without the intermediate plan, and report whether the CoDes advantage survives. In addition, if plan samples are repeated, the quantity labeled Pass@1 should be described as Pass@1 after plan retries rather than standard Pass@1.","section":"6.1; Tables 1–2; Appendices D, E.1, H"},{"comment":"The abstract and Section 6.1 state that CoDes 'significantly surpasses' standard prompting, but no confidence intervals, bootstrap errors, or paired significance tests are provided. On a 202-problem benchmark, a Pass@1 difference of 3–6 percentage points corresponds to roughly 6–12 additional passing problems and may be within sampling noise. Please add per-problem paired tests (e.g., McNemar's test) or bootstrap confidence intervals for the Table 1 versus Table 2 comparisons, and avoid the word 'significantly' unless a test supports it.","section":"Section 6.1; Tables 1 and 2"},{"comment":"The Single-step versus Multi-step ablation is internal to CoDes and does not resolve the confound identified above: both conditions appear to use the same plan-formulation stage, so the comparison isolates execution strategy rather than the value of descriptive plans relative to standard prompting. The text also says Multi-step 'significantly improves' performance without reporting uncertainty; please clarify which settings are shared between Single-step and Multi-step and add the same statistical support.","section":"Section 6.4; Figure 2 (right)"}],"minor_comments":[{"comment":"The second row repeats '% of Type-2 Clones'; this should presumably read '% of Type-3 Clones'.","section":"Table 3"},{"comment":"The summation uses index i while the text defines the evaluation set by j; please make the notation consistent.","section":"Section 4.2.2, Eq. (1)"},{"comment":"There is a typo in 'summarizae'; it should be 'summarize'.","section":"Appendix F"},{"comment":"In the left judge prompt, SUMMARY-2 is written as '“{Reference”}' with an unbalanced quote; please fix the quoting.","section":"Figure 4"},{"comment":"The paper alternates between 'Chain-of-Description' and 'Chain-of-Descriptions', and between 'CoDes' and 'chain-of-description' (Appendix H); please standardize the terminology.","section":"Throughout"},{"comment":"The relationship between the plan-formulation settings (T=0.89, top-k=50) and the final execution sampling settings is not stated; please specify the decoding settings used for the final code or summary generation in each condition.","section":"Appendix D; Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The VHDL-Eval benchmark is cited as an anonymous in-press work [2], and several central tables depend on it; please ask the authors to provide the full citation or public dataset details so the evaluation can be reproduced. The paper otherwise fits the scope of MLCAD, but the main empirical comparison must be re-run with matched baselines before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the contribution here is the VHDL-Xform dataset and the zero-shot evaluation grid, not the CoDes method as an established result. The CoDes gains are consistent across all eight models, but the headline comparison in Tables 1 and 2 is confounded by sampling budget and decoding hyperparameters, so the size of the benefit from the intermediate descriptive steps is not actually measured.\n\nWhat is new and worth keeping: VHDL-Xform, a 6,500-pair clone-transformation dataset targeting functional equivalence; a careful zero-shot evaluation of eight LLMs on VHDL generation, with testbench and sequential equivalence checking as verification; and a plan-then-execute prompt strategy adapted to VHDL. SEC is a strong verification step, and the pattern of improvement over the baseline is uniform, which hints that CoDes does something real.\n\nThe soft spot is the one the stress-test flags, and it is load-bearing. Appendix D states the baseline uses a single sample at T=0.5 or 0.7 with top-k=40; Appendix E.1 and H reveal that the plan formulation uses T=0.89, top-k=50, and retries up to three times when the plan does not match the expected extraction pattern. Table 2 further uses Multi-step execution, meaning at least two LLM calls (plan, then execute) versus one call for the baseline. So the comparison does not isolate the descriptive steps. The paper also says 'significantly surpasses' without any significance test or confidence interval. That is a major gap for the central claim.\n\nSmaller issues: reference [2] is anonymous, so the VHDL-Eval side of the evaluation cannot be independently checked, and the in-house dataset is not released, which blocks reproduction. The limitation section is candid about the retry behavior but never acknowledges that it breaks the controlled comparison.\n\nAudience: researchers working on LLMs for hardware description languages. The dataset and evaluation grid will be useful to them; the CoDes result is a plausible preliminary finding, not a proven one. I would bring this to a reading group as a case study in confounded prompt-comparison methodology.\n\nRecommendation: send it to peer review, but direct the authors to repeat the main comparison with matched sampling (same temperature, top-k, number of calls) and report error bars. If the gain survives, the paper is solid; if not, the dataset and evaluation still justify publication.","headline":"Useful VHDL benchmark and evaluation, but the CoDes effectiveness claim is confounded by unmatched sampling budgets and needs a controlled redo before being taken at face value.","tokens_in":15115,"tokens_out":3445,"would_cite":true,"duration_ms":35946,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-Descriptions lifts VHDL outputs across code LLMs.","keywords":["VHDL code generation","VHDL code summarization","Chain-of-Descriptions","code LLMs","Electronic Design Automation","self-consistency score","sequential equivalence checking","RTL code synthesis"],"falsifier":"Run CoDes under exactly the same sampling conditions as the zero-shot baseline (single sample, temperature 0.5 or 0.7, top-k=40) and independently give the baseline the same three-sample budget; if the Pass@1 gap disappears, the descriptive steps are not the causal driver.","tokens_in":14032,"feed_emoji":"🧠","tokens_out":4938,"duration_ms":56837,"temperature":0.7,"pith_summary":"This paper claims that large language models are weak at VHDL, a hardware description language central to chip design, and that a prompting strategy called Chain-of-Descriptions (CoDes) substantially narrows that gap. CoDes asks the model first to produce intermediate descriptive steps—a natural-language plan for code generation, or a line-by-line explanation for summarization—then appends that plan to the original prompt and asks for the final output. Testing eight LLMs on two datasets, the paper reports that CoDes outperforms plain zero-shot prompting on most metrics, with Granite-Code-34b's Pass@1 on self-verifying testbenches rising from 0.192 to 0.254. The authors read this as evidence that VHDL's difficulty for LLMs is partly a planning problem that structured descriptions can mitigate, and they offer CoDes as a benchmark for future VHDL-focused work.","feed_headline":"Descriptive plans lift LLM VHDL code scores across models","feed_subtitle":"CoDes raises Granite-Code-34b's Pass@1 from 0.192 to 0.254 on VHDL testbenches.","key_machinery":"The central mechanism is the Chain-of-Descriptions (CoDes) pipeline: a planning prompt that elicits a series of intermediate descriptive steps from the LLM, a regex-based refinement step that removes boilerplate and normalizes the steps, and an execution step that combines the refined plan with the original input to produce the final code or summary. For generation, the plan prompt is 'Generate a plan specifying the intermediate steps to <problem>'; for summarization, the model is prompted line-by-line, with an Abstract Syntax Tree alternative for larger programs. Multi-step execution, where planning and final generation are separate prompts, outperforms single-step execution, indicating the plan is doing active work in shaping the output.","core_discovery":"When asked directly for VHDL code, LLMs tend to guess at an abstraction level that misses the requested functionality—for instance, generating a fixed 4-bit register while the problem asks for an n-bit register. CoDes prevents this by decomposing the task: formulate a descriptive plan, refine it into clean numbered steps, then execute by feeding the refined plan back to the model alongside the original prompt. The paper reports that this multi-step formulation improves Pass@1 on the VHDL-Eval dataset for every model family tested, with Granite-Code-34b rising from 0.192 to 0.254 on self-verifying testbenches and from 0.187 to 0.246 under sequential equivalence checking. Summarization metrics—LLM preference rate and ROUGE-L—also improve on the in-house VHDL-Xform dataset. Longer problem descriptions help more than short ones, and multi-step execution beats single-step execution, which the authors take as evidence that the intermediate descriptions themselves carry the benefit.","pith_inferences":["The paper does not compare CoDes against a zero-shot baseline given the same sampling budget; a matched-compute ablation would clarify whether the gains come from the descriptive steps or from the extra sampling effort used during plan formulation.","The same planning-plus-execution recipe could plausibly transfer to other hardware description languages such as Verilog and SystemVerilog, though the paper only evaluates VHDL.","For real-world RTL designs, line-by-line summarization plans will become expensive at scale; the paper's AST-based alternative points toward a scalable direction that future work could strengthen.","The judge-LLM preference metric could itself be a source of variance; replacing it with direct simulation or equivalence checking would make the summarization claims more self-contained."],"forward_implications":["Existing code LLMs are not reliable for VHDL: the best zero-shot testbench Pass@1 in the paper is 0.192, far below their performance on mainstream languages.","Under CoDes, every model tested improves on VHDL-Eval code generation Pass@1, with Granite-based models gaining roughly 40% relative improvement on testbench correctness.","CoDes also improves VHDL summarization as measured by LLM preference rate and ROUGE-L on both VHDL-Eval and VHDL-Xform.","Longer descriptive prompts and multi-step execution each boost results, suggesting that decomposition and plan quality are the active ingredients.","The framework is task-agnostic in structure and can serve as a baseline methodology for future VHDL-LLM benchmarks and model development."],"supporting_citations":[{"why":"Supplies the VHDL-Eval dataset, including canonical solutions and self-verifying testbenches used for code generation evaluation and self-consistency.","marker":"[2]"},{"why":"Provides the sequential equivalence checking methodology used to verify functional correctness in the SEC Pass@1 metric.","marker":"[3]"},{"why":"Defines the Pass@k evaluation metric that the paper adapts for VHDL code generation.","marker":"[6]"},{"why":"The Verilog-Eval benchmark whose problems were translated to create VHDL-Eval.","marker":"[10]"},{"why":"Introduces the IdentityChain evaluation adapted here as the self-consistency score SC1.","marker":"[11]"},{"why":"Provides the code-clone taxonomy that motivates the Type-2, Type-3, and Type-4 transformations in the VHDL-Xform dataset.","marker":"[16]"},{"why":"Chain-of-thought prompting is the conceptual antecedent for the intermediate descriptive-step planning in CoDes.","marker":"[22]"},{"why":"Supplies the LLM preference-rate method used to judge the quality of generated code summaries.","marker":"[23]"}],"fun_headline_variants":["CoDes: descriptive steps lift LLM VHDL generation and summarization","CoDes improves LLM VHDL coding and summarization scores","LLMs write better VHDL with Chain-of-Descriptions steps","Descriptive planning boosts LLM VHDL generation and summarization","CoDes: LLMs rise on VHDL generation and summarization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim assumes that the score improvements under CoDes come from the descriptive plan itself, not from the fact that the plan step uses different sampling settings and up to three attempts to produce a valid plan.","fun_headline_variants_meta":{"raw":{"variants":["CoDes: descriptive steps lift LLM VHDL generation and summarization","CoDes improves LLM VHDL coding and summarization scores","LLMs write better VHDL with Chain-of-Descriptions steps","Descriptive planning boosts LLM VHDL generation and summarization","CoDes: LLMs rise on VHDL generation and summarization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000728,"raw_usage":{"total_tokens":3323,"prompt_tokens":1068,"completion_tokens":2255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":2165}},"tokens_in":684,"tokens_out":2255,"duration_ms":18761,"temperature":1.0,"reasoning_tokens":2165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:48:16.304159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoDes under exactly the same sampling conditions as the zero-shot baseline (single sample, temperature 0.5 or 0.7, top-k=40) and independently give the baseline the same three-sample budget; if the Pass@1 gap disappears, the descriptive steps are not the causal driver.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VHDL-Eval dataset, including canonical solutions and self-verifying testbenches used for code generation evaluation and self-consistency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sequential equivalence checking methodology used to verify functional correctness in the SEC Pass@1 metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Verilog-Eval benchmark whose problems were translated to create VHDL-Eval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the code-clone taxonomy that motivates the Type-2, Type-3, and Type-4 transformations in the VHDL-Xform dataset."}],"review_version":1}