Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces FIXME, the first end-to-end, multi-modal, and quantitative evaluation framework for LLM-aided hardware functional verification, and finds that GPT-4o leads with a 39.58% average functional pass rate while all models…

desk verdict A genuinely useful benchmark artifact with a misleading TD metric that drives the headline number; the qualitative story likely survives a fix. read the letter →

arxiv 2507.04276 v1 pith:UZHEOKOV submitted 2025-07-06 cs.AR

Gwok-Waa Wan , Shengchu Su , Ruihu Wang , Qixiang Chen , Sam-Zaak Wong , Mengnv Xing , Hefei Feng , Yubo Wang
show 10 more authors
Yinan Zhu Jingyi Zhang Jianmin Ye Xinlai Wan Tao Ni Qiang Xu Nan Guan Zhe Jiang Xi Wang Yang Jun
This is my paper
classification cs.AR
keywords LLM-aideddesignfunctionalverificationhardwarebenchmarktestbenchgenerationSystemVerilogassertionstestcaseLLMevaluationsilicon-provendesigns
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FIXME is presented as the first end-to-end, multi-modal, quantitative evaluation framework for LLM-aided hardware functional verification. The benchmark spans six verification sub-domains and 180 tasks built from silicon-proven OpenCores designs, organized into three difficulty levels. Evaluating six state-of-the-art LLMs, the paper reports that GPT-4o achieves the strongest overall functional pass rate of 39.58%, while testcase design is a consistent weak point across all models, averaging 7.72%. The authors argue this fills a gap left by code-generation benchmarks and provides a reusable, open-source evaluation platform.

What carries the argument

The framework's carrying mechanism is a six-sub-domain task taxonomy mapped one-to-one to the functional verification flow: specification comprehension (multiple choice), reference model generation (coding), testcase design (short answer), testbench generation (coding), assertion generation (coding), and debugging (coding). Each sub-domain has a dedicated metric, most notably a semantic-completeness score for testcase design that applies 60% and 90% thresholds to vector-based similarity with expert reference answers, and line/toggle coverage for testbenches. A three-level difficulty hierarchy based on lines of code and expert-adjusted complexity, plus a multi-agent data pipeline (VerifyAgent) with human refinement that raised functional coverage by 45.57%, complete the mechanism.

What would settle it

A reader could run the model-generated testcases from the TD tasks against the benchmark DUTs with injected faults and compute the correlation between the assigned similarity scores and the actual bug-detection rate; a correlation near zero would falsify the metric and the 7.72% finding.

Watch

Extended reading notes

Core claim

The central claim is that current LLMs can handle syntax-level verification tasks and basic specification comprehension, but fail at deep semantic tasks such as testcase design and SVA generation. The paper quantifies this through FIXME's six sub-domains, reporting GPT-4o as the best overall performer at 39.58% average functional pass rate, Claude3 with the most robust testbench coverage at 68%, and all models averaging only 7.72% on testcase design. This is positioned as the first benchmark covering the entire functional verification workflow rather than isolated generation or formal-verification stages.

Load-bearing premise

The testcase-design score assumes that semantic similarity to an expert reference answer measures a testcase's real ability to exercise the design or catch bugs, yet the paper never tests that correlation.

Editorial extensions

If this is right

  • If the measured 39.58% top average is representative, no current LLM can be handed an end-to-end verification task and expected to converge without human intervention.
  • The low 7.72% testcase-design average implies that models good at generating testbenches and assertions still cannot identify which functional features actually need testing.
  • The benchmark's open 180-task dataset gives the community a fixed yardstick for comparing future LLMs on verification rather than generation.
  • The 45.57% coverage gain from expert enhancement indicates that real-world verification benchmarks cannot be built from raw open-source RTL alone.
  • The observed decline in pass rates with difficulty level suggests the three-level hierarchy is effective at probing model capability boundaries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The semantic-similarity metric for testcase design could be validated head-to-head against a simulation-based oracle: run generated testcases on DUTs with seeded bugs and check whether similarity predicts bug detection.
  • The VerifyAgent pipeline might transfer to other data-scarce hardware domains, such as UVM environment generation or formal property mining, where the same scarcity and quality problems apply.
  • The paper's pass@5 protocol with up to three feedback iterations leaves open how much of the reported performance comes from iteration; isolating the feedback effect would sharpen comparisons.
  • The 45.57% coverage improvement suggests a benchmark-construction recipe, but also implies that FIXME's tasks may be harder than typical LLM training data, possibly explaining low scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FIXME, an open-source benchmark of 180 hardware functional-verification (FV) tasks spanning six sub-domains (specification comprehension, reference model generation, testcase design, testbench generation, assertion generation, and debugging) at three difficulty levels, built from OpenCores designs through an AI-human pipeline. Six LLMs are evaluated, and the paper reports that GPT-4o has the strongest overall verification capability with an average functional pass rate of 39.58%, that Claude3 achieves the best testbench coverage, and that testcase design is weak across all models (average 7.72%). The paper also claims a 45.57% improvement in functional coverage from expert-guided optimization and states that the dataset uses 100% silicon-proven designs. The authors position FIXME as the first end-to-end, multi-modal, quantitative evaluation framework for LLM-aided design verification.

Significance. If corrected, FIXME would fill a real gap: existing LLM hardware benchmarks concentrate on RTL generation, while verification is arguably the more costly and less benchmarked stage. The released 180-task benchmark with difficulty levels and the qualitative finding that LLMs are stronger at syntax-level verification tasks than at semantic tasks such as testcase design are valuable and plausible. The use of an external simulator (Synopsys VCS) as ground truth for the code-generation tasks (MG, TG, AG, Debug) is methodologically appropriate and avoids circularity; the fact that the VerifyAgent pipeline helped draft tasks is not, by itself, a problem, since the reference answers were expert-refined and code tasks were judged by simulation. The main quantitative claims, however, currently rest on a semantic-similarity proxy for testcase design and on internally inconsistent dataset descriptions, so the headline numbers cannot be used as stated until those issues are fixed.

major comments (4)
  1. [III.F, Eq. (2), Table III, IV.B] The TD sub-domain is scored by vector-based semantic similarity to an expert reference answer using arbitrary thresholds (below 60% gives zero, 60-90% gives proportional credit, above 90% counts as success). This is not a functional pass rate: it has not been validated against whether a testcase actually exercises the target behavior or catches a bug. Table III nevertheless labels this row 'Func. PR', and Section IV.B uses it to support the headline claims of a 39.58% average functional pass rate and the statement that testcase design is weak (average 7.72%). Moreover, the continuous similarity score is incompatible with Eq. (2), which defines pass rate by exact equality of predicted and reference answers. Finally, the GPT-4o average of 39.58% does not cleanly reproduce from the six row averages in Table III (the simple mean is about 38.8%), so the aggregation rule needs to be stated. Please relabel the TD metric, validate or recalibrate it against a functional criterion, and recompute all averages that include it.
  2. [Abstract vs. III.E] The abstract and contribution list state that the dataset is constructed 'using 100% silicon-proven designs,' but Section III.E says that 'more than one-third of the projects have undergone FPGA or silicon verification.' These statements are contradictory. The dataset's claimed real-world validity depends on this point, so the paper must either provide evidence that all selected modules are silicon-proven or revise the claim to match the actual selection criteria.
  3. [III.E.3 and Section I contribution bullet] The paper reports an average 'functional coverage' improvement of 45.57%, but the same paragraph reports an average line coverage increase of 7.5% and a toggle coverage increase of 43.14%. Averaging line and toggle coverage gains does not yield a functional coverage metric; functional coverage in verification is normally defined by coverage points tied to design features, not by line and toggle statistics. If 'functional coverage' is used in the headline, it must be defined and measured directly; otherwise the coverage contribution bullet should report line and toggle coverage separately.
  4. [IV.A and Table III] Section IV.A states that 'For all models tested, we performed a pass@5 statistical analysis,' but the paper never defines how pass@5 is computed or how it is mapped to the percentages in Table III and Figure 5. If pass@5 is an estimate of the probability that at least one of five generations passes, it is not the same as Eq. (2)'s per-task pass rate, and the reported averages are ambiguous. The paper should state the estimator used (e.g., an unbiased pass@k estimator) and clarify whether Table III reports pass@1, pass@5, or the mean over generated samples.
minor comments (5)
  1. [III.E.3] The phrase 'rigorous audit of 50K lines of design code and choice of 15K of them' is unclear; it should say that 15K lines were selected from 50K audited lines.
  2. [IV.A] The model list in Section IV.A refers to 'Claude-3.5-Haiku-200k' and elsewhere to 'Claude3'; the naming should be made consistent throughout.
  3. [V.A.1] The claim that pass rates 'exhibit a declining pattern' with difficulty is not universally supported: for example, in Table III the TG row for LLaMA3 increases from L1 to L2 (28.76% to 31.23%) and for Mistral-Large increases from 61.23% to 71.24%. The discussion should acknowledge non-monotonic cases or report a statistical test of the difficulty gradient.
  4. [References] Reference [45] for Semikong is incomplete: the arXiv identifier is given as 'arXiv:2024.xxxxx'. This must be replaced with the correct citation.
  5. [Table III] The column header 'Func. PR' is used for all rows, but the SC row reports a plain pass rate and the TD row reports a semantic-similarity score; the table should use distinct metric names in rows or a note explaining the different definitions.

Circularity Check

1 steps flagged · score 6.0 of 10

TD sub-domain 'functional pass rate' is a semantic-similarity score by construction, making the headline 'testcase design is weak' finding an artifact of the chosen scoring rule.

  1. fitted input called prediction [Section III.F (Criteria Metric); Table III; Section IV.B; Section V.A.3]
    "FIXME introduces an assessment based on semantic completeness for the TD sub-domain’s short-answer questions. This metric compares the generated content with reference answers by leveraging vector-based semantic similarity. Scores are assigned according to the similarity level: if the similarity is below 60%, the testcase is considered a failure and assigned a score of zero; if the similarity falls between 60% and 90%, the score is proportional to the similarity percentage; if the similarity exceeds 90%, the testcase is deemed successful."

    Equation (2) defines pass rate as exact equality between predicted and reference answers, but the TD score is a continuous vector-similarity measure with author-chosen thresholds. Table III nonetheless reports this TD score under the column header 'Func. PR', and Section IV.B interprets it as 'the design of testcases remains a weak area for all the evaluated models' (average 7.72%, Section V.A.3). The low TD 'functional pass rate' is therefore not a simulation-based verification result; it is the semantic-similarity value relabeled as a functional pass rate. No evidence is given that similarity to an expert reference text predicts whether a testcase exercises the DUT or detects a bug. The headline TD weakness is thus entailed by the scoring construction rather than measured independently.

full rationale

Most of FIXME's evaluation is externally anchored: MG, TG, AG, and Debug scores are determined by Synopsys VCS simulation of generated RTL/testbenches/assertions, and SC is objective multiple-choice, so those dimensions do not reduce to the models' own outputs. The self-citations (ChatCPU [2], MEIC [4], VGV [44]) are contextual and not load-bearing. The circular component is confined to the TD sub-domain: Section III.F defines TD scoring as vector-based semantic similarity to expert reference answers with arbitrary thresholds, while Table III labels these scores 'Func. PR' and Section IV.B/V.A treat the resulting 7.72% average as evidence that testcase design is a weak area. Because the TD 'functional pass rate' is by construction the similarity score, this particular headline finding is an artifact of the chosen metric and thresholds, not an independent verification measurement. This is partial circularity: it affects one of six sub-domains and the aggregate average, but the remaining evaluation pipeline retains external simulation ground truth. Non-circular correctness concerns (e.g., unvalidated TD threshold, abstract's '100% silicon-proven' vs. Section III.E's 'more than one-third') are noted but do not change the circularity finding.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new theoretical entities (particles, forces, dimensions, conserved quantities) are introduced. VerifyAgent is a software pipeline of LLM agents, a concrete implementation artifact with an open-source handle, not a postulated entity. The two free parameters are hand-chosen thresholds that shape the most fragile metric (TD) and the difficulty hierarchy.

free parameters (2)
  • TD semantic-similarity thresholds = 0.60 fail / 0.90 pass cutoffs
    Section III.F assigns testcase-design scores by similarity: below 60% scores zero, 60-90% scores proportionally, above 90% passes. The cutoffs are arbitrary and never calibrated against whether a testcase would actually exercise the design or catch a bug.
  • LOC difficulty tier boundaries = 0-100 / 100-200 / >200 lines
    Section III.E defines the three-level difficulty hierarchy by lines of code, subsequently adjusted by human experts; the cutoffs are hand-chosen and the expert adjustment is not quantified.
assumptions (3)
  • domain assumption The OpenCores project pool, after automated VCS smoke-test filtering, is representative of real-world functional verification scenarios and difficulty.
    Section III.D-E narrows data collection to OpenCores and retains only projects whose existing testbenches compile and pass under VCS; the benchmark's external validity and the difficulty-gradation claims rest on this representativeness, which is asserted, not demonstrated.
  • ad hoc to paper Vector-based semantic similarity to an expert reference answer measures testcase design quality.
    Section III.F defines TD scoring by similarity thresholds (60%/90%) and never validates the score against whether the testcase exercises the design or catches a bug; this metric drives the paper's finding that testcase design is the weakest area.
  • ad hoc to paper Line and toggle coverage gains can be aggregated into a 'functional coverage' improvement of 45.57%.
    Section III.E reports line coverage +7.5% and toggle coverage +43.14%, but the contributions and abstract state 'functional coverage increased by an average of 45.57%' with no stated aggregation rule; (7.5+43.14)/2 is 25.3% and 7.5+43.14 is 50.64%, so 45.57% is not recoverable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification." pith.science (2026). https://pith.science/paper/UZHEOKOV

@misc{pith2026250704276,
  author       = {Pith},
  title        = {Pith review of: FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZHEOKOV}},
  note         = {Machine review of arXiv:2507.04276}
}
read the original abstract

Despite the transformative potential of Large Language Models (LLMs) in hardware design, a comprehensive evaluation of their capabilities in design verification remains underexplored. Current efforts predominantly focus on RTL generation and basic debugging, overlooking the critical domain of functional verification, which is the primary bottleneck in modern design methodologies due to the rapid escalation of hardware complexity. We present FIXME, the first end-to-end, multi-model, and open-source evaluation framework for assessing LLM performance in hardware functional verification (FV) to address this crucial gap. FIXME introduces a structured three-level difficulty hierarchy spanning six verification sub-domains and 180 diverse tasks, enabling in-depth analysis across the design lifecycle. Leveraging a collaborative AI-human approach, we construct a high-quality dataset using 100% silicon-proven designs, ensuring comprehensive coverage of real-world challenges. Furthermore, we enhance the functional coverage by 45.57% through expert-guided optimization. By rigorously evaluating state-of-the-art LLMs such as GPT-4, Claude3, and LlaMA3, we identify key areas for improvement and outline promising research directions to unlock the full potential of LLM-driven automation in hardware design verification. The benchmark is available at https://github.com/ChatDesignVerification/FIXME.

Figures

Figures reproduced from arXiv: 2507.04276 by the authors.

Figure 1
Figure 1. Traditional Design Functional Verification [9] [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Top Reasons for Silicon Failed [11] [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Framework of FIXME Construction and Evaluation. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Evaluations of Tested Models on FIXME TABLE III RESULTS OF THE PART OF FUNCTIONAL PASS RATE. GIVEN THE SPACE CONSTRAINTS, WE HAVE SELECTED A COMMON EVALUATION METRIC ACROSS THE DIFFERENT SUB-DOMAINS OF THE FIXME TO PRESENT THE DATA AND PROVIDE THE NECESSARY BASIS FOR A…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Revolution or Hype? Seeking the Limits of Large Models in Hardware Design

    cs.LG 2025-09 conditional novelty 1.0 of 10

    Large models can help early-stage hardware design and verification, but their reliability, data, and precision limits mean traditional EDA algorithms and formal verification remain necessary.

Reference graph

Works this paper leans on

45 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models

    Yonggan Fu, Yongan Zhang, Zhongzhi Yu, Sixu Li, Zhifan Ye, Chaojian Li, Cheng Wan, and Yingyan Celine Lin. Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) , pages 1–9. IEEE, 2023

    work page 2023

  2. [2]

    Chatcpu: An agile cpu design & verification platform with llm

    Xi Wang, Gwok-Waa Wan, Sam-Zaak Wong, Layton Zhang, Tianyang Liu, Qi Tian, and Jianmin Ye. Chatcpu: An agile cpu design & verification platform with llm. In 61st ACM/IEEE Design Automation Conference (DAC’24), page 6, 2024

    work page 2024

  3. [3]

    (security) assertions by large language models

    Rahul Kande, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Shailja Thakur, Ramesh Karri, and Jeyavijayan Rajendran. (security) assertions by large language models. IEEE Transactions on Information Forensics and Security, 19:4374–4389, 2024

    work page 2024

  4. [4]

    Meic: Re-thinking rtl debug automation using llms

    Ke Xu, Jialin Sun, Yuchen Hu, Xinwei Fang, Weiwei Shan, Xi Wang, and Zhe Jiang. Meic: Re-thinking rtl debug automation using llms. In 2024 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2024

    work page 2024

  5. [5]

    Verilogeval: Evaluating large language models for verilog code genera- tion

    Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code genera- tion. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pages 1–8. IEEE, 2023

    work page 2023

  6. [6]

    Llm4sechw: Leveraging domain-specific large language model for hardware debugging

    Weimin Fu, Kaichen Yang, Raj Gautam Dutta, Xiaolong Guo, and Gang Qu. Llm4sechw: Leveraging domain-specific large language model for hardware debugging. In 2023 Asian Hardware Oriented Security and Trust Symposium (AsianHOST) , pages 1–6, 2023

    work page 2023

  7. [7]

    What’s missing in agile hardware design? verification! J

    Babak Falsafi. What’s missing in agile hardware design? verification! J. Comput. Sci. Technol. , 38(4):735–736, 2023

    work page 2023

  8. [8]

    H ¨am¨al¨ainen

    Sakari Lahti, Panu Sj ¨ovall, Jarno Vanne, and Timo D. H ¨am¨al¨ainen. Are we there yet? a study on the state of high-level synthesis. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 38(5):898–911, 2019

    work page 2019

Show all 45 references
  1. [9]

    Delivering Functional Verification Engagements

    Synopsys. Delivering Functional Verification Engagements. Synopsys White Paper, 2015

    2015

  2. [10]

    Chip-chat: Challenges and opportunities in conversational hardware design

    Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. Chip-chat: Challenges and opportunities in conversational hardware design. In 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD), pages 1–6, 2023

    2023

  3. [11]

    Directed test generation for hardware validation: A survey

    Aruna Jayasena and Prabhat Mishra. Directed test generation for hardware validation: A survey. ACM Comput. Surv. , 56(5), January 2024

    2024

  4. [12]

    Electronic Design Automation: Synthesis, Verification, and Test

    Laung-Terng Wang, Yao-Wen Chang, and Kwang-Ting (Tim) Cheng. Electronic Design Automation: Synthesis, Verification, and Test. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2009

    2009

  5. [13]

    Formal and simulation verification: Comparing and contrasting the two verification approaches

    M Girish, G Gopakumar, and D S Divya. Formal and simulation verification: Comparing and contrasting the two verification approaches. In 2021 2nd International Conference on Advances in Computing, Communication, Embedded and Secure Systems (ACCESS) , pages 41– 44, 2021

    2021

  6. [14]

    Functional verification 2003: technology, tools and methodology

    Carl Pixley, Aruna Chittor, Fred Meyer, Steve McMaster, and Dan Benua. Functional verification 2003: technology, tools and methodology. In ASIC, 2003. Proceedings. 5th International Conference on, volume 1, pages 1–5. IEEE, 2003

    2003

  7. [15]

    William K. Lam. Hardware Design Verification: Simulation and Formal Method-Based Approaches. Prentice Hall PTR, USA, 1st edition, 2008

    2008

  8. [16]

    Greenstreet

    Christoph Kern and Mark R. Greenstreet. Formal verification in hardware design: a survey. ACM Trans. Des. Autom. Electron. Syst. , 4(2):123–193, 1999

    1999

  9. [17]

    RTL Formal Verification, pages 103–129

    Springer US. RTL Formal Verification, pages 103–129. Springer US, Boston, MA, 2001

    2001

  10. [18]

    Towards Developing High Performance RISC-V Processors Using Agile Methodology

    Yinan Xu, Zihao Yu, Dan Tang, Guokai Chen, Lu Chen, Lingrui Gou, Yue Jin, Qianruo Li, Xin Li, et al. Towards Developing High Performance RISC-V Processors Using Agile Methodology. In MICRO, 2022

    2022

  11. [19]

    High-level synthesis, volume 1

    Philippe Coussy and Adam Morawiec. High-level synthesis, volume 1. Springer, 2010

    2010

  12. [20]

    System-on-a-chip verification: methodology and techniques

    Prakash Rashinkar, Peter Paterson, and Leena Singh. System-on-a-chip verification: methodology and techniques. Kluwer Academic Publishers, USA, 2001

    2001

  13. [21]

    Verigen: A large language model for verilog code generation

    Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. Verigen: A large language model for verilog code generation. ACM Trans. Des. Autom. Electron. Syst. , 29(3), 2024

    2024

  14. [22]

    BetterV: Controlled verilog generation with discriminative guidance

    Zehua Pei, Huiling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. BetterV: Controlled verilog generation with discriminative guidance. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 40145–401...

    2024

  15. [23]

    Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution

    Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution. In 2024 IEEE LLM Aided Design Workshop (LAD) , pages 1–5, 2024

    2024

  16. [24]

    Rtlfixer: Automatically fixing rtl syntax errors with large language model

    Yunda Tsai, Mingjie Liu, and Haoxing Ren. Rtlfixer: Automatically fixing rtl syntax errors with large language model. In Proceedings of the 61st ACM/IEEE Design Automation Conference , DAC ’24, New York, NY , USA, 2024

    2024

  17. [25]

    Chipgpt: How far are we from natural language hardware design

    Kaiyan Chang, Ying Wang, Haimeng Ren, Mengdi Wang, Shengwen Liang, Yinhe Han, Huawei Li, and Xiaowei Li. Chipgpt: How far are we from natural language hardware design. arXiv preprint arXiv:2305.14019, 2023

    2023

  18. [26]

    LLM4DV: Using large language models for hardware test stimuli generation

    Zixi Zhang, Greg Chadwick, Hugo McNally, Yiren Zhao, and Robert Mullins. LLM4DV: Using large language models for hardware test stimuli generation. In Machine Learning for Systems 2023 , 2023

    2023

  19. [27]

    Verilogreader: Llm-aided hardware test gener- ation

    Ruiyang Ma, Yuxin Yang, Ziqian Liu, Jiaxi Zhang, Min Li, Junhua Huang, and Guojie Luo. Verilogreader: Llm-aided hardware test gener- ation. In 2024 IEEE LLM Aided Design Workshop (LAD) , pages 1–5, 2024

    2024

  20. [28]

    Assertllm: Generating hardware verification assertions from design specifications via multi-llms

    Wenji Fang, Mengming Li, Min Li, Zhiyuan Yan, Shang Liu, Hongce Zhang, and Zhiyao Xie. Assertllm: Generating hardware verification assertions from design specifications via multi-llms. In 2024 IEEE LLM Aided Design Workshop (LAD) , pages 1–1, 2024

    2024

  21. [29]

    Autobench: Automatic testbench generation and evaluation using llms for hdl design

    Ruidi Qiu, Grace Li Zhang, Rolf Drechsler, Ulf Schlichtmann, and Bing Li. Autobench: Automatic testbench generation and evaluation using llms for hdl design. In Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD , MLCAD ’24, New York, NY , USA, 2024

    2024

  22. [30]

    Rtllm: An open- source benchmark for design rtl generation with large language model

    Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. Rtllm: An open- source benchmark for design rtl generation with large language model. In Proceedings of the 29th Asia and South Pacific Design Automation Conference, ASPDAC ’24, page 722–727. IEEE Press, 2024

    2024

  23. [31]

    Fveval: Understanding language model capabilities in for- mal verification of digital hardware

    Minwoo Kang, Mingjie Liu, Ghaith Bany Hamad, Syed Suhaib, and Haoxing Ren. Fveval: Understanding language model capabilities in for- mal verification of digital hardware. arXiv preprint arXiv:2410.23299 , 2024

    2024 arXiv

  24. [32]

    Specification and design of embed- ded hardware-software systems

    Daniel D Gajski and Frank Vahid. Specification and design of embed- ded hardware-software systems. IEEE Design & Test of Computers , 12(1):53–67, 1995

    1995

  25. [33]

    Behavioral con- sistency of c and verilog programs using bounded model checking

    Edmund Clarke, Daniel Kroening, and Karen Yorav. Behavioral con- sistency of c and verilog programs using bounded model checking. In Proceedings of the 40th annual Design Automation Conference , pages 368–371, 2003

    2003

  26. [34]

    Digital design with RTL design, VHDL, and Verilog

    Frank Vahid. Digital design with RTL design, VHDL, and Verilog . John Wiley & Sons, 2010

    2010

  27. [35]

    Writing testbenches using SystemVerilog

    Janick Bergeron. Writing testbenches using SystemVerilog . Springer Science & Business Media, 2007

    2007

  28. [36]

    SystemVerilog for verification: a guide to learning the testbench language features

    Chris Spear. SystemVerilog for verification: a guide to learning the testbench language features. Springer Science & Business Media, 2008

    2008

  29. [37]

    Assertion-based design

    Harry D Foster, Adam C Krolnik, and David J Lacey. Assertion-based design. Springer Science & Business Media, 2004

    2004

  30. [38]

    Cadre: Cycle- accurate deterministic replay for hardware debugging

    Smruti R Sarangi, Brian Greskamp, and Josep Torrellas. Cadre: Cycle- accurate deterministic replay for hardware debugging. In International Conference on Dependable Systems and Networks (DSN’06), pages 301–

  31. [39]

    Chipnemo: Domain-adapted llms for chip design, 2024

    Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Desh- pande,...

    2024

  32. [40]

    https://opencores.org/

    Opencores. https://opencores.org/. Accessed: 2024-10-20

    2024

  33. [41]

    Let me do it for you: Towards llm empowered rec- ommendation via tool learning

    Yuyue Zhao, Jiancan Wu, Xiang Wang, Wei Tang, Dingxian Wang, and Maarten de Rijke. Let me do it for you: Towards llm empowered rec- ommendation via tool learning. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval...

    2024

  34. [42]

    Flexkbqa: A flexible llm-powered frame- work for few-shot knowledge base question answering

    Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, and Jianyong Wang. Flexkbqa: A flexible llm-powered frame- work for few-shot knowledge base question answering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 1860...

    2024

  35. [43]

    Natural language is not enough: Benchmarking multi-modal generative ai for verilog generation, 2024

    Kaiyan Chang, Zhirong Chen, Yunhao Zhou, Wenlong Zhu, kun wang, Haobo Xu, Cangyuan Li, Mengdi Wang, Shengwen Liang, Huawei Li, Yinhe Han, and Ying Wang. Natural language is not enough: Benchmarking multi-modal generative ai for verilog generation, 2024

    2024

  36. [44]

    Vgv: Verilog generation using visual capabilities of multi-modal large lan- guage models

    Sam-Zaak Wong, Gwok-Waa Wan, Dongping Liu, and Xi Wang. Vgv: Verilog generation using visual capabilities of multi-modal large lan- guage models. In 2024 IEEE LLM Aided Design Workshop (LAD) , pages 1–5, 2024

    2024

  37. [45]

    Semikong: Curating, training, and evaluating a semiconductor industry-specific large language model

    Christopher Nguyen et al. Semikong: Curating, training, and evaluating a semiconductor industry-specific large language model. arXiv preprint arXiv:2024.xxxxx, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.