REVIEW 3 major objections 4 minor 46 references
Large Language Models (LLMs) for Electronic Design Automation (EDA)
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read LLMs can take on real chip-design chores, this survey argues, and three case studies show how.
desk verdict A useful but uneven survey of LLM-for-EDA; the only quantitative case study is too weak to support the abstract's automation claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The unifying mechanism is the alignment between hardware description languages and the textual formats LLMs are trained on. The load-bearing technical pieces in the case studies are: (1) retrieval-augmented generation with a repair template library, verified by C-RTL co-simulation, for HLS repair; (2) backward slicing plus spectra monitoring to focus LLM-generated test inputs on behavioral discrepancies; and (3) structural chain-of-thought prompting inside an evolutionary candidate-pool loop, with temperature adaptation and Levenshtein-distance diversity pressure, for system-level test generation. Each case study closes the loop with a deterministic EDA tool (HLS compiler, simulator, FPGA po
What would settle it
Run the three case-study pipelines, with code and prompts made public, on hardware tasks outside the authors' group—e.g., HLS repair of a different benchmark suite, testbench generation for an open RISC-V core, and SLT power generation for a different processor—and measure whether LLM output is usable without human rewriting.
Extended reading notes
Core claim
The core claim is that because HDL code, HLS C/C++, and test programs are text, LLMs can serve as general-purpose assistants across the entire chip design-to-manufacturing flow, from specification to physical implementation and test. The paper presents three case studies: (i) LLM-aided repair of C/C++ programs into HLS-compatible form, using retrieval-augmented generation plus equivalence verification; (ii) LLM-driven testbench adaptation and behavioral-discrepancy testing for HLS, using backward slicing, spectra monitoring, and LLM-guided test-input generation; and (iii) an evolutionary loop that uses structural chain-of-thought prompting and temperature adaptation to generate C snippets th
Load-bearing premise
The paper's general conclusion assumes that the three case studies, all drawn from the authors' own prior work, are representative of what LLMs can do across EDA rather than selected successes.
Editorial extensions
If this is right
- HLS designers could submit ordinary C/C++ and have LLM repair convert it to synthesizable code, reducing manual rewriting.
- Testbench generation and behavioral discrepancy testing could be partially automated, lowering verification effort.
- System-level test programs could be generated automatically to stress non-functional conditions like power, without deep microarchitecture knowledge.
- The feedback-loop pattern—LLM proposes, EDA tool scores, loop repeats—could generalize to other EDA tasks such as synthesis pragma tuning or layout script generation.
- A unified multi-modal agent spanning specification, RTL, netlist, and layout is a plausible next target, though the paper marks it as an open challenge.
Reading between the lines
- The strongest generalization of the survey is likely the closed-loop pattern, not any single case study; the three cases are all tasks where a deterministic tool can score the LLM's output, which is exactly where LLM errors are catchable. For tasks without a cheap oracle, automation gains may be far smaller.
- The SLT power-maximization approach seems directly transferable to other non-functional targets—temperature, switching activity, or instruction throughput—and to other processors, since the loop only needs a measurable score.
- A testable extension: feed the same evolutionary loop with assembly-level snippets as a baseline; the paper's own comparison with genetic programming suggests the LLM advantage may be in producing realistic high-level code, not in raw optimization power.
- If the field adopts this survey's framing, the next bottleneck will be benchmark diversity: progress claims will need standard suites that are not drawn from the proposing group's own tools.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This special-session paper surveys recent applications of large language models (LLMs) across the electronic design automation (EDA) flow, from specification and HDL generation through verification, high-level synthesis (HLS) repair, logic synthesis, physical design, and system-level test (SLT). It presents three case studies drawn largely from the authors' prior work: an LLM-aided C/C++ program repair framework for HLS (Section III), an LLM-guided behavioral-discrepancy testing framework for HLS (Section III), and an LLM-driven optimization loop that generates power-maximizing C code for SLT on a RISC-V processor (Section V). The paper concludes that integrating LLMs into EDA is a 'transformative opportunity' that can simplify and even automate the entire workflow, and it lists future directions such as bridging semantic gaps, expert-level HLS, and multi-modal EDA agents.
Significance. If the empirical claims held, the paper would offer a useful structured map of a fast-moving area and a concrete demonstration that LLMs can generate nontrivial hardware artifacts. The paper has several strengths: clear figures, a detailed algorithmic description of the SLT optimization loop, and an explicit acknowledgment that the LLM result was not better than the genetic-programming baseline in the single reported comparison. It also cites a broad set of recent papers. However, the central claim rests on very thin quantitative evidence: the only numerical case study is a single unreplicated optimization run with unequal compute, the fine-tuning claim is unquantified, and the two HLS frameworks are presented without evaluation metrics. The paper is therefore more persuasive as a position statement and qualitative survey than as an empirical demonstration of automation.
major comments (3)
- [Section V (SLT case study)] The quantitative core of the paper is one LLM optimization run (24 h, 2021 snippets, best 5.042 W) compared with one GP run (39 h, best 5.682 W). There are no error bars, no multiple seeds, and the runtimes are unequal; the text even notes that GP kept improving after 24 h. This cannot support the claim that the LLM approach is 'promising' in a measurable sense. The sentence 'Compared to the off-the-shelf model, it performs significantly better' is also unsupported: no numbers, test set, or significance test are given. Since this case study is the only quantitative evidence for the abstract's automation promise, please either add replication/statistics or explicitly reframe the result as a single anecdotal data point, and report the fine-tuning comparison quantitatively or remove the 'significantly better' claim.
- [Section III (HLS repair and discrepancy testing)] The two HLS frameworks are described in algorithmic detail, but no evaluation results are reported: no repair success rate, no number of benchmarks, no baseline comparison for HLS-Repair, and no coverage or discrepancy-detection rate for HLSTester. Because these are two of the three case studies invoked to support the paper's central claim, their inclusion as 'demonstrations' does not provide evidence. Please either add the key quantitative results (or point to the specific numbers in the original papers) or clearly label these sections as workflow illustrations rather than empirical demonstrations.
- [Section II (State of the Art)] The paper describes itself as a 'comprehensive overview,' but the survey is a selective narrative without a search protocol, inclusion criteria, or comparative evaluation tables. Strong comparative statements, e.g., 'The CodeGen models outperformed ChatGPT-3.5 and performed similarly well to GPT-4' and 'the most capable model ... was the only one to significantly benefit from feedback,' are given without benchmark numbers or confidence information. This does not invalidate the survey, but it makes the 'comprehensive' characterization hard to verify and weakens the authority of the conclusions. Please add a comparative summary table with available metrics or qualify the scope of the overview.
minor comments (4)
- [References [21], [25]] Reference [25] (MCP4EDA) lists the same arXiv identifier (2309.09437) as reference [21]; this is likely a typo and should be corrected.
- [Figure 1] The rendered text in Figure 1 is garbled in places (e.g., 'LLSM [2' and the floating 'Compiler Log' box). Please check the figure source and caption for readability.
- [Section V] The temperature-adaptation schedule and the number of random examples per prompt (n) are key free parameters, but no formulas or settings are given in the text; the reader is referred to [37]. Consider giving at least the functional form or a small parameter table.
- [Abstract and Conclusion] The abstract and conclusion repeatedly state that LLMs can 'simplify and even automate the entire workflow.' Given the evidence in Section V, we recommend softening the wording to 'may help simplify' or 'show potential to simplify' unless additional experimental support is added.
Circularity Check
No circular derivation; case studies are self-citations but not reduction-to-input.
full rationale
The paper is a survey/special-session overview, not a derivation chain. The central claim (LLMs can simplify/automate EDA) is supported by a literature review and three case studies, two from prior work by the same authors [28],[30] and one from [37]. No equation or result is defined in terms of another result; no fitted parameter is renamed a prediction. The SLT comparison (Section V: 'Our optimization loop ran for 24 hours and produced 2021 snippets. The best snippet in this run consumes 5.042 W' vs a 39-hour GP run at 5.682 W) is an unreplicated empirical comparison, but the paper does not claim to have predicted this number from the inputs; it explicitly calls out runtime mismatch and multiple plausible causes. The statement that the fine-tuned Code Llama 'performs significantly better' lacks quantitative support, and the HLS frameworks in Section III are described without success rates; these are correctness/evidence weaknesses, not circularity. Self-citations are prevalent, but they cite published prior work as examples and outlook, not as an unexamined premise that forces the conclusion. No 'uniqueness theorem' is imported and no ansatz is smuggled in via citation. Therefore no specific circular step can be identified; the low score reflects only the paper's heavy reliance on the authors' own prior studies as illustrative evidence.
Assumptions & free parameters
free parameters (2)
- LLM temperature adaptation schedule
- Number of random examples per prompt (n)
assumptions (2)
- domain assumption C-RTL co-simulation correctly verifies functional equivalence of repaired HLS code.
- domain assumption FPGA power measurement is a valid proxy for marginal defect detection in system-level test.
Cite this review
Pith. "Pith review of Large Language Models (LLMs) for Electronic Design Automation (EDA)." pith.science (2026). https://pith.science/paper/UASJCE5U
@misc{pith2026250820030,
author = {Pith},
title = {Pith review of: Large Language Models (LLMs) for Electronic Design Automation (EDA)},
year = {2026},
howpublished = {\url{https://pith.science/paper/UASJCE5U}},
note = {Machine review of arXiv:2508.20030}
}
read the original abstract
With the growing complexity of modern integrated circuits, hardware engineers are required to devote more effort to the full design-to-manufacturing workflow. This workflow involves numerous iterations, making it both labor-intensive and error-prone. Therefore, there is an urgent demand for more efficient Electronic Design Automation (EDA) solutions to accelerate hardware development. Recently, large language models (LLMs) have shown remarkable advancements in contextual comprehension, logical reasoning, and generative capabilities. Since hardware designs and intermediate scripts can be represented as text, integrating LLM for EDA offers a promising opportunity to simplify and even automate the entire workflow. Accordingly, this paper provides a comprehensive overview of incorporating LLMs into EDA, with emphasis on their capabilities, limitations, and future opportunities. Three case studies, along with their outlook, are introduced to demonstrate the capabilities of LLMs in hardware design, testing, and optimization. Finally, future directions and challenges are highlighted to further explore the potential of LLMs in shaping the next-generation EDA, providing valuable insights for researchers interested in leveraging advanced AI technologies for EDA.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Machine Learning in Advanced IC Design: A Methodological Survey,
Tinghuan Chen, Grace Li Zhang, Bei Yu, Bing Li, Ulf Schlichtmann, “Machine Learning in Advanced IC Design: A Methodological Survey,” IEEE Design & Test, 2023
work page 2023
-
[2]
Chip-Chat: Chal- lenges and Opportunities in Conversational Hardware Design,
Jason Blocklove, Siddharth Garg, Ramesh Karri, Hammond Pearce, “Chip-Chat: Chal- lenges and Opportunities in Conversational Hardware Design,” MLCAD, 2023
work page 2023
-
[3]
Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design,
Andre Nakkab, Sai Qian Zhang, Ramesh Karri, Siddharth Garg, “Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design,” MLCAD, 2024
work page 2024
-
[4]
Benchmarking Large Language Models for Automated Verilog RTL Code Generation,
Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, Siddharth Garg, “Benchmarking Large Language Models for Automated Verilog RTL Code Generation,” DATE, 2023
work page 2023
-
[5]
Large Language Models for Verification, Testing, and Design,
Chandan Kumar Jha et al., “Large Language Models for Verification, Testing, and Design,” ETS, 2025
work page 2025
-
[6]
LLM-Aided Efficient Hardware Design Automation,
K. Xu et al., “LLM-Aided Efficient Hardware Design Automation,” arXiv:2410.18582
-
[7]
SpecLLM: Exploring Generation and Review of VLSI Design Specification with Large Language Model,
Mengming Li et al., “SpecLLM: Exploring Generation and Review of VLSI Design Specification with Large Language Model,” arXiv preprint: 2401.13266, 2024
arXiv 2024
-
[8]
Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback,
J. Blocklove, S. Thakur, B. Tan, H. Pearce, S. Garg, and R. Karri, “Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback,” TODAES, 2025
work page 2025
Show all 46 references
-
[9]
GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models,
Yonggan Fu, Yongan Zhang, Zhongzhi Yu, Sixu Li, Zhifan Ye, Chaojian Li, Cheng Wan, Yingyan Lin, “GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models,” ICCAD, 2023
2023
-
[10]
Evaluating LLMs for Hardware Design and Test,
Jason Blocklove, Siddharth Garg, Ramesh Karri, Hammond Pearce, “Evaluating LLMs for Hardware Design and Test,” LAD, 2024
2024
-
[11]
Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA,
Yuan Pu et al., “Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA,” ICCAD, 2024
2024
-
[12]
FPGA HLS Today: Successes, Challenges, and Opportunities,
Jason Cong et al., “FPGA HLS Today: Successes, Challenges, and Opportunities,” TRETS, 2022
2022
-
[13]
New solutions on LLM acceleration, optimization, and application,
Yingbing Huang, Lily Jiaxin Wan, Hanchen Ye, Manvi Jha, Jinghua Wang, Yuhong Li, Xiaofan Zhang, Deming Chen, “New solutions on LLM acceleration, optimization, and application,” DAC, 2024
2024
-
[14]
VRank: Enhancing Verilog Code Generation from Large Lan- guage Models via Self-Consistency,
Zhuorui Zhao et al., “VRank: Enhancing Verilog Code Generation from Large Lan- guage Models via Self-Consistency,” ISQED, 2025
2025
-
[15]
VFocus: Better Verilog Generation from Large Language Model via Focused Reasoning,
Zhuorui Zhao et al., “VFocus: Better Verilog Generation from Large Language Model via Focused Reasoning,” SOCC, 2025
2025
-
[16]
RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models,
Yun-Da Tsai et al., “RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models,” arXiv preprint: 2311.16543, 2023
2023 arXiv
-
[17]
C2HLSC: Leveraging Large Language Models to Bridge the Software-to-Hardware Design Gap,
Luca Collini, Siddharth Garg, Ramesh Karri, “C2HLSC: Leveraging Large Language Models to Bridge the Software-to-Hardware Design Gap,” TODAES, 2025
2025
-
[18]
AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design,
Ruidi Qiu et al., “AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design,” MLCAD, 2024
2024
-
[19]
CorrectBench: Automatic Testbench Generation with Functional Self- correction Using LLMs for HDL Design,
Ruidi Qiu et al., “CorrectBench: Automatic Testbench Generation with Functional Self- correction Using LLMs for HDL Design,” DATE, 2025
2025
-
[20]
AssertLLM: Generating Hardware Verification Assertions from Design Speci- fications via Multi-LLMs,
Wenji Fang, Mengming Li, Min Li, Zhiyuan Yan, Shang Liu, Zhiyao Xie, Hongce Zhang, “AssertLLM: Generating Hardware Verification Assertions from Design Speci- fications via Multi-LLMs,” IEEE LLM Aided Design Workshop (LAD), 2024
2024
-
[22]
LLSM: LLM-enhanced Logic Synthesis Model with EDA-guided CoT Prompting,
Shan Huang et al., “LLSM: LLM-enhanced Logic Synthesis Model with EDA-guided CoT Prompting,” ASP-DAC, 2024
2024
-
[23]
Advanced LLM-Driven Verilog Development: Enhancing Power, Performance, and Area Optimization in Code Synthesis,
Kiran Thorat et al., “Advanced LLM-Driven Verilog Development: Enhancing Power, Performance, and Area Optimization in Code Synthesis,” arXiv: 2312.01022, 2023
2023 arXiv
-
[24]
VeriOpt: PPA-Aware High-Quality Verilog Generation via Multi- Role LLMs,
Kimia Tasnia et al., “VeriOpt: PPA-Aware High-Quality Verilog Generation via Multi- Role LLMs,” ICCAD, 2025
2025
-
[25]
MCP4EDA: LLM-Powered Model Context Protocol RTL-to-GDSII Automation with Backend Aware Synthesis Optimization,
Yiting Wang, Wanghao Ye, Yexiao He, Yiran Chen, Gang Qu, Ang Li, “MCP4EDA: LLM-Powered Model Context Protocol RTL-to-GDSII Automation with Backend Aware Synthesis Optimization,” arXiv preprint: 2309.09437, 2025
2025 arXiv
-
[26]
LayoutCopilot: An LLM-powered Multi-agent Collaborative Framework for Interactive Analog Layout Design,
Bingyang Liu, Haoyi Zhang, Xiaohan Gao, Zichen Kong, Xiyuan Tang, Yibo Lin, Run- sheng Wang, Ru Huang, “LayoutCopilot: An LLM-powered Multi-agent Collaborative Framework for Interactive Analog Layout Design,” ISCAS, 2025
2025
-
[27]
HLSRewriter: Efficient Refactoring and Optimization of C/C++ Code with LLMs for High-Level Synthesis,
Kangwei Xu et al., “HLSRewriter: Efficient Refactoring and Optimization of C/C++ Code with LLMs for High-Level Synthesis,” TODAES, 2024
2024
-
[28]
Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models,
Kangwei Xu et al., “Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models,” MLCAD, 2024
2024
-
[29]
Logic Design of Neural Networks for High-Throughput and Low- Power Applications,
Kangwei Xu et al., “Logic Design of Neural Networks for High-Throughput and Low- Power Applications,” ASP-DAC, 2024
2024
-
[30]
HLSTester: Efficient Testing of Behavioral Discrepancies with LLMs for High-Level Synthesis,
Kangwei Xu et al., “HLSTester: Efficient Testing of Behavioral Discrepancies with LLMs for High-Level Synthesis,” ICCAD, 2025
2025
-
[31]
Fault Testing and Diagnosis Techniques for Carbon Nanotube-Based FPGAs,
K. Xu et al., “Fault Testing and Diagnosis Techniques for Carbon Nanotube-Based FPGAs,” ASP-DAC, 2022
2022
-
[32]
Exploring the Mysteries of System-Level Test,
I. Polian et al., “Exploring the Mysteries of System-Level Test,” ATS, 2020
2020
-
[33]
Beyond structural test, the rising need for system-level test,
H. H. Chen, “Beyond structural test, the rising need for system-level test,” VLSI-DAT, 2018
2018
-
[34]
Automating Greybox System-Level Test Generation,
D. Schwachhofer, M. Betka, S. Becker, S. Wagner, M. Sauer, and I. Polian, “Automating Greybox System-Level Test Generation,” ETS, 2023
2023
-
[35]
Optimizing System-Level Test Program Generation via Genetic Programming,
D. Schwachhofer et al., “Optimizing System-Level Test Program Generation via Genetic Programming,” ETS, 2024
2024
-
[36]
SonicBOOM: The 3rd Generation Berkeley Out-of-Order Machine,
J. Zhao et al., “SonicBOOM: The 3rd Generation Berkeley Out-of-Order Machine,” Fourth Workshop on Computer Architecture Research with RISC-V , 2020
2020
-
[37]
Large Language Model-based Optimization for System-Level Test Program Generation,
D. Schwachhofer et al., “Large Language Model-based Optimization for System-Level Test Program Generation,” DFT, 2024
2024
-
[38]
Structured Chain-of-Thought Prompting for Code Generation,
J. Li, G. Li, Y . Li, and Z. Jin, “Structured Chain-of-Thought Prompting for Code Generation,” arXiv: arXiv:2305.06599, 2023
2023 arXiv
-
[39]
DA VE: Deriving Automatically Verilog from English,
H. Pearce, B. Tan, and R. Karri, “DA VE: Deriving Automatically Verilog from English,” MLCAD, 2020
2020
-
[40]
VeriGen: A Large Language Model for Verilog Code Generation,
S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri, and S. Garg, “VeriGen: A Large Language Model for Verilog Code Generation,” TODAES, 2024
2024
-
[41]
RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation Technique,
S. Liu et al., “RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation Technique,” TCAD, 2024
2024
-
[42]
VerilogEval: Evaluating Large Language Models for Verilog Code Generation,
M. Liu et al., “VerilogEval: Evaluating Large Language Models for Verilog Code Generation,” ICCAD, 2023
2023
-
[43]
CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarization,
Y . Zhao et al., “CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarization,” arXiv:2407.10424, 2025
2025 arXiv
-
[44]
LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trust,
Z. Wang et al., “LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trust,” ISVLSI, 2024
2024
-
[45]
Verileaky: Navigating IP Protection vs. Utility in Fine-Tuning for LLM- Driven Verilog Coding,
Z. Wang et al., “Verileaky: Navigating IP Protection vs. Utility in Fine-Tuning for LLM- Driven Verilog Coding,” ICLAD, 2025
2025
-
[46]
RTL-Breaker: Assessing the Security of LLMs Against Backdoor Attacks on HDL Code Generation,
L. L. Mankali et al., “RTL-Breaker: Assessing the Security of LLMs Against Backdoor Attacks on HDL Code Generation,” DATE, 2025
2025
-
[47]
SALAD: Systematic Assessment of Machine Unlearning on LLM- Aided Hardware Design,
Z. Wang et al., “SALAD: Systematic Assessment of Machine Unlearning on LLM- Aided Hardware Design,” MLCAD, 2025
2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.