Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-Descriptions lifts VHDL outputs across code LLMs.

desk verdict Useful VHDL benchmark and evaluation, but the CoDes effectiveness claim is confounded by unmatched sampling budgets and needs a controlled redo before being taken at face value. read the letter →

arxiv 2507.12308 v1 pith:PE5PPVPM submitted 2025-07-16 cs.CL cs.AIcs.AR

Prashanth Vijayaraghavan , Apoorva Nitsure , Charles Mackin , Luyao Shi , Stefano Ambrogio , Arvind Haran , Viresh Paruthi , Ali Elzein
show 4 more authors
Dan Coops David Beymer Tyler Baldwin Ehsan Degan
This is my paper
classification cs.CLcs.AIcs.AR
keywords VHDLcodegenerationsummarizationChain-of-DescriptionsLLMsElectronicDesignAutomationself-consistencyscoresequentialequivalencecheckingRTLsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language models are weak at VHDL, a hardware description language central to chip design, and that a prompting strategy called Chain-of-Descriptions (CoDes) substantially narrows that gap. CoDes asks the model first to produce intermediate descriptive steps—a natural-language plan for code generation, or a line-by-line explanation for summarization—then appends that plan to the original prompt and asks for the final output. Testing eight LLMs on two datasets, the paper reports that CoDes outperforms plain zero-shot prompting on most metrics, with Granite-Code-34b's Pass@1 on self-verifying testbenches rising from 0.192 to 0.254. The authors read this as evidence that VHDL's difficulty for LLMs is partly a planning problem that structured descriptions can mitigate, and they offer CoDes as a benchmark for future VHDL-focused work.

What carries the argument

The central mechanism is the Chain-of-Descriptions (CoDes) pipeline: a planning prompt that elicits a series of intermediate descriptive steps from the LLM, a regex-based refinement step that removes boilerplate and normalizes the steps, and an execution step that combines the refined plan with the original input to produce the final code or summary. For generation, the plan prompt is 'Generate a plan specifying the intermediate steps to <problem>'; for summarization, the model is prompted line-by-line, with an Abstract Syntax Tree alternative for larger programs. Multi-step execution, where planning and final generation are separate prompts, outperforms single-step execution, indicating the plan is doing active work in shaping the output.

What would settle it

Run CoDes under exactly the same sampling conditions as the zero-shot baseline (single sample, temperature 0.5 or 0.7, top-k=40) and independently give the baseline the same three-sample budget; if the Pass@1 gap disappears, the descriptive steps are not the causal driver.

Watch

Extended reading notes

Core claim

When asked directly for VHDL code, LLMs tend to guess at an abstraction level that misses the requested functionality—for instance, generating a fixed 4-bit register while the problem asks for an n-bit register. CoDes prevents this by decomposing the task: formulate a descriptive plan, refine it into clean numbered steps, then execute by feeding the refined plan back to the model alongside the original prompt. The paper reports that this multi-step formulation improves Pass@1 on the VHDL-Eval dataset for every model family tested, with Granite-Code-34b rising from 0.192 to 0.254 on self-verifying testbenches and from 0.187 to 0.246 under sequential equivalence checking. Summarization metrics—LLM preference rate and ROUGE-L—also improve on the in-house VHDL-Xform dataset. Longer problem descriptions help more than short ones, and multi-step execution beats single-step execution, which the authors take as evidence that the intermediate descriptions themselves carry the benefit.

Load-bearing premise

The paper's central claim assumes that the score improvements under CoDes come from the descriptive plan itself, not from the fact that the plan step uses different sampling settings and up to three attempts to produce a valid plan.

Editorial extensions

If this is right

  • Existing code LLMs are not reliable for VHDL: the best zero-shot testbench Pass@1 in the paper is 0.192, far below their performance on mainstream languages.
  • Under CoDes, every model tested improves on VHDL-Eval code generation Pass@1, with Granite-based models gaining roughly 40% relative improvement on testbench correctness.
  • CoDes also improves VHDL summarization as measured by LLM preference rate and ROUGE-L on both VHDL-Eval and VHDL-Xform.
  • Longer descriptive prompts and multi-step execution each boost results, suggesting that decomposition and plan quality are the active ingredients.
  • The framework is task-agnostic in structure and can serve as a baseline methodology for future VHDL-LLM benchmarks and model development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not compare CoDes against a zero-shot baseline given the same sampling budget; a matched-compute ablation would clarify whether the gains come from the descriptive steps or from the extra sampling effort used during plan formulation.
  • The same planning-plus-execution recipe could plausibly transfer to other hardware description languages such as Verilog and SystemVerilog, though the paper only evaluates VHDL.
  • For real-world RTL designs, line-by-line summarization plans will become expensive at scale; the paper's AST-based alternative points toward a scalable direction that future work could strengthen.
  • The judge-LLM preference metric could itself be a source of variance; replacing it with direct simulation or equivalence checking would make the summarization claims more self-contained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper evaluates eight code, instruct, and chat LLMs on two VHDL benchmarks: VHDL-Eval, a 202-problem benchmark with self-verifying testbenches and reference solutions, and VHDL-Xform, a newly introduced in-house dataset of 6,500 code-clone pairs used to probe functional-equivalence understanding. The evaluation reports Pass@1 using both testbenches and sequential equivalence checking (SEC), a self-consistency score, ROUGE-L, and an LLM judge-based preference rate. After documenting consistent underperformance of existing LLMs, the authors propose Chain-of-Descriptions (CoDes), which first prompts the LLM to generate intermediate descriptive steps for a problem statement or code snippet, post-processes these steps into a plan, and then feeds the refined plan together with the original input to the LLM to produce the final code or summary. Table 2 reports consistent improvements for CoDes over the zero-shot baseline, and Section 6 presents ablations on prompt length, AST-based versus line-based planning, and single-step versus multi-step execution.

Significance. The paper makes useful contributions to an under-studied area: it introduces VHDL-Xform, evaluates a broader set of LLMs than most prior HDL work, and uses both self-verifying testbenches and formal SEC to assess functional correctness rather than relying on compile-only checks. The self-consistency metric and the judge-LLM preference rate are also reasonable evaluation ideas. However, the central quantitative claim that CoDes 'significantly surpasses' standard prompting is not yet established, because the CoDes condition is not matched to the baseline in sampling budget, decoding hyperparameters, retry behavior, or number of LLM calls. If a matched-control experiment confirms that the descriptive plan itself drives the gains, this framework would be a practical and reproducible contribution to LLM-assisted VHDL design; as it stands, the main comparison is confounded.

major comments (3)
  1. [6.1; Tables 1–2; Appendices D, E.1, H] The headline comparison between zero-shot and CoDes is not matched in sampling behavior. Appendix D states that zero-shot results use a single sample at T=0.5 or 0.7 with top-k=40, whereas the CoDes plan formulation uses T=0.89, top-k=50, and up to three sampling attempts before accepting a parseable plan (Appendices E.1 and H), and Multi-Step execution adds a second LLM call (Section 5.3.2). Consequently the reported Pass@1 improvements (for example, Granite-Code-34b from 0.192 to 0.254 on testbenches and from 0.187 to 0.246 on SEC) may be caused by the larger sampling budget, the different temperature, the retry-until-extract mechanism, or the extra inference compute, rather than by the descriptive plan itself. This concern is load-bearing because the paper's central claim is that the CoDes plan structure improves performance. Please add a control in which the baseline receives exactly the same decoding hyperparameters, the same number of sampling attempts, and the same number of LLM calls without the intermediate plan, and report whether the CoDes advantage survives. In addition, if plan samples are repeated, the quantity labeled Pass@1 should be described as Pass@1 after plan retries rather than standard Pass@1.
  2. [Section 6.1; Tables 1 and 2] The abstract and Section 6.1 state that CoDes 'significantly surpasses' standard prompting, but no confidence intervals, bootstrap errors, or paired significance tests are provided. On a 202-problem benchmark, a Pass@1 difference of 3–6 percentage points corresponds to roughly 6–12 additional passing problems and may be within sampling noise. Please add per-problem paired tests (e.g., McNemar's test) or bootstrap confidence intervals for the Table 1 versus Table 2 comparisons, and avoid the word 'significantly' unless a test supports it.
  3. [Section 6.4; Figure 2 (right)] The Single-step versus Multi-step ablation is internal to CoDes and does not resolve the confound identified above: both conditions appear to use the same plan-formulation stage, so the comparison isolates execution strategy rather than the value of descriptive plans relative to standard prompting. The text also says Multi-step 'significantly improves' performance without reporting uncertainty; please clarify which settings are shared between Single-step and Multi-step and add the same statistical support.
minor comments (6)
  1. [Table 3] The second row repeats '% of Type-2 Clones'; this should presumably read '% of Type-3 Clones'.
  2. [Section 4.2.2, Eq. (1)] The summation uses index i while the text defines the evaluation set by j; please make the notation consistent.
  3. [Appendix F] There is a typo in 'summarizae'; it should be 'summarize'.
  4. [Figure 4] In the left judge prompt, SUMMARY-2 is written as '“{Reference”}' with an unbalanced quote; please fix the quoting.
  5. [Throughout] The paper alternates between 'Chain-of-Description' and 'Chain-of-Descriptions', and between 'CoDes' and 'chain-of-description' (Appendix H); please standardize the terminology.
  6. [Appendix D; Section 5.3] The relationship between the plan-formulation settings (T=0.89, top-k=50) and the final execution sampling settings is not stated; please specify the decoding settings used for the final code or summary generation in each condition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CoDes evaluation is grounded in external testbenches and SEC oracles, and no fitted parameter or self-referential definition drives the central claim.

full rationale

The paper's central claim is empirical: CoDes improves Pass@1 and other metrics over zero-shot prompting. The final VHDL code is verified either by self-verifying testbenches or by sequential equivalence checking against the canonical solution, and summaries are scored by ROUGE-L and an external judge LLM. No parameter is fitted to minimize error on the test sets, and no equation defines the predicted output in terms of the input in a way that forces the result. The plan generated by the LLM is fed back into the prompt, but the final response is still checked against external oracles, so there is no definitional reduction. The self-consistency metric reuses the model's own generated summary to generate a new program; this is a defined measurement of round-trip consistency, not a derivation of a target result from its own assumptions. The reliance on the anonymous VHDL-Eval reference [2] is a provenance and verifiability concern, not a circularity concern. The main legitimate weakness is experimental: the CoDes condition uses different sampling hyperparameters (T=0.89, top-k=50), up to three plan-generation retries, and an extra LLM call relative to the zero-shot baseline, which makes the improvement not fully attributable to the descriptive steps. However, that is a confound in experimental design, not a case where the prediction reduces by construction to a fitted input or a self-citation chain. Under the stated rules, this does not constitute circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of three evaluation tools: SEC for functional equivalence, the judge LLM for summary preference, and the VHDL-Xform transformations for semantic-preserving clones. None of these are fitted parameters; the only hand-chosen settings are the decoding hyperparameters for plan generation, which differ between baseline and CoDes and are a potential confound.

free parameters (1)
  • CoDes plan formulation settings (T=0.89, top-k=50, up to 3 samples) = T=0.89; top-k=50; samples=3
    Appendix D: CoDes uses these decoding settings for plan formulation, whereas the zero-shot baseline uses T=0.5/0.7, top-k=40, single sample. These choices are not justified as matched and could inflate CoDes results.
assumptions (4)
  • domain assumption SEC correctly checks functional equivalence between generated VHDL and the reference canonical solution.
    Used in Pass@1 (SEC) and self-consistency (Sections 4.2.1 and 4.2.2 of the paper). If SEC has toolchain bugs or the canonical solutions are not truly correct, the numbers are unreliable.
  • domain assumption The VHDL-Xform transformations preserve functionality while creating code clones.
    Section 3.2 defines Type-2, Type-3, and Type-4 transformations. If any transformation alters behavior, then summaries of transformed code cannot be compared with original summaries as tests of functional equivalence.
  • domain assumption Llama-3-70B judge LLM yields valid preference judgments for summary quality.
    Section 4.2.4 and Appendix E: The PR metric is computed by this one judge. If the judge is biased toward certain summary styles, the PR improvements may not reflect true quality.
  • domain assumption The VHDL-Eval dataset problems are representative of VHDL coding tasks relevant to practice.
    The dataset has 202 textbook-style problems scraped from tutorials and translated from Verilog-Eval (Appendix A). Appendix H acknowledges real designs are more complex; the conclusions may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization." pith.science (2026). https://pith.science/paper/PE5PPVPM

@misc{pith2026250712308,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PE5PPVPM}},
  note         = {Machine review of arXiv:2507.12308}
}
read the original abstract

Large Language Models (LLMs) have become widely used across diverse NLP tasks and domains, demonstrating their adaptability and effectiveness. In the realm of Electronic Design Automation (EDA), LLMs show promise for tasks like Register-Transfer Level (RTL) code generation and summarization. However, despite the proliferation of LLMs for general code-related tasks, there's a dearth of research focused on evaluating and refining these models for hardware description languages (HDLs), notably VHDL. In this study, we evaluate the performance of existing code LLMs for VHDL code generation and summarization using various metrics and two datasets -- VHDL-Eval and VHDL-Xform. The latter, an in-house dataset, aims to gauge LLMs' understanding of functionally equivalent code. Our findings reveal consistent underperformance of these models across different metrics, underscoring a significant gap in their suitability for this domain. To address this challenge, we propose Chain-of-Descriptions (CoDes), a novel approach to enhance the performance of LLMs for VHDL code generation and summarization tasks. CoDes involves generating a series of intermediate descriptive steps based on: (i) the problem statement for code generation, and (ii) the VHDL code for summarization. These steps are then integrated with the original input prompt (problem statement or code) and provided as input to the LLMs to generate the final output. Our experiments demonstrate that the CoDes approach significantly surpasses the standard prompting strategy across various metrics on both datasets. This method not only improves the quality of VHDL code generation and summarization but also serves as a framework for future research aimed at enhancing code LLMs for VHDL.

Figures

Figures reproduced from arXiv: 2507.12308 by the authors.

Figure 1
Figure 1. (a) Overview of the Chain-of-Description (CoDes) framework: Input prompts and final responses vary between code [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Results of Ablation Studies: Left: Short vs. Long Descriptive prompts for plan formulation for Code Generation Task [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Samples of different transformation strategies applied using the three categories of code clones – Type 2, Type 3 and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Prompt used for LLM judgment – Left: Prompt for computing the preference rate for VHDL-Eval dataset. Right: [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Single-Step vs. Multi-Step Plan Execution Strategies [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 9 canonical work pages

  1. [2]

    Anonymous. 2024. VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation. In Press. Proceedings of the First IEEE Interna- tional Workshop on LLM-Aided Design (2024)

    work page 2024

  2. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Jason Baumgartner, Hari Mony, Viresh Paruthi, Robert Kanzelman, and Geert Janssen. 2006. Scalable sequential equivalence checking across arbitrary design transformations. In 2006 International Conference on Computer Design . IEEE, 259–266

    work page 2006

  4. [4]

    Stefan Bellon, Rainer Koschke, Giulio Antoniol, Jens Krinke, and Ettore Merlo

    work page

  5. [5]

    Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2023. Chip-chat: Challenges and opportunities in conversational hardware design. In 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD) . IEEE, 1–6

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  7. [7]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023)

    arXiv 2023

  8. [8]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

    arXiv 2023

Show all 24 references
  1. [9]

    Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. 2023. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176 (2023)

    2023 arXiv

  2. [10]

    Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023. Ver- ilogeval: Evaluating large language models for verilog code generation. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) . IEEE, 1–8

    2023

  3. [11]

    Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2023. Beyond accuracy: Evaluating self-consistency of code large language models with identitychain. arXiv preprint arXiv:2310.14053 (2023)

    2023 arXiv

  4. [12]

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. 2024. Granite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:...

    2024 arXiv

  5. [13]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  6. [14]

    Hammond Pearce, Benjamin Tan, and Ramesh Karri. 2020. Dave: Deriving automatically verilog from english. InProceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD . 27–32

    2020

  7. [15]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  8. [16]

    Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu, Pierre Baldi, and Cristina V Lopes. 2018. Oreo: Detection of clones in the twilight zone. In Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of softw...

    2018

  9. [17]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  10. [18]

    Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. 2023. Benchmarking large language models for automated verilog rtl code generation. In 2023 Design, Automation & Test in Europe Conference & Exhib...

    2023

  11. [19]

    Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan- Gavitt, Ramesh Karri, and Siddharth Garg. 2024. Verigen: A large language model for verilog code generation. ACM Transactions on Design Automation of Electronic Systems 29, 3 (2024), 1–31

    2024

  12. [20]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  13. [21]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    2022 arXiv

  14. [22]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  15. [23]

    Create a VHDL code for computing a logical OR of all the bits in the input

    Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. 2023. Evaluating instruction-tuned large language models on code compre- hension and generation. arXiv preprint arXiv:2308.01240 (2023). A VHDL-Eval Dataset VHDL-Eval benchmark is designed for eval...

    2023 arXiv

  16. [2007]

    IEEE Transactions on software engineering 33, 9 (2007), 577–591

    Comparison and evaluation of clone detection tools. IEEE Transactions on software engineering 33, 9 (2007), 577–591

    2007

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.