Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RealBench claims to be the first benchmark for real-world IP-level Verilog generation, with handwritten multi-modal specifications and formal equivalence checking, and it finds that current LLMs pass at most 13.3% of module-level tasks…

desk verdict RealBench is a genuinely harder, better-verified Verilog benchmark whose main conclusion is robust, though the module-level formal-equivalence criterion is stricter than the spec and deserves transparency. read the letter →

arxiv 2507.16200 v1 pith:EJLQE5FP submitted 2025-07-22 cs.LG cs.AR

Pengwei Jin , Di Huang , Chongxiao Li , Shuyao Cheng , Yang Zhao , Xinyao Zheng , Jiaguo Zhu , Shuyi Xing
show 5 more authors
Bohan Dou Rui Zhang Zidong Du Qi Guo Xing Hu
This is my paper
classification cs.LGcs.AR
keywords VeriloggenerationhardwaredesignautomationLLMbenchmarkformalequivalencecheckingtestbenchlinecoverageRTLcodesubmoduleinstantiationmulti-modalspecifications
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RealBench is presented as the first benchmark for real-world IP-level Verilog generation, built from four open-source IP designs with handwritten, multi-modal specifications and testbenches that reach 100% line coverage plus formal equivalence checking. The paper's central empirical finding is that current LLMs are far from ready for this task: the best model tested, o1-preview, achieves 13.3% formal@1 on module-level tasks and 0% on system-level tasks, with other models scoring lower. The paper also shows that earlier benchmarks overstate ability, reporting that 44.2% of code that passes RTLLMV2's testbenches fails formal equivalence checking. The authors argue that only a benchmark combining realistic design complexity, formatted specifications, and rigorous verification can measure actual progress in automated chip design.

What carries the argument

The object that carries the argument is the verification pipeline, not any single theorem. Each task supplies a handwritten, hierarchical, multi-modal design specification intended to be sufficient for a competent engineer to implement the module from scratch, and success is judged by two gates: a manually built testbench that achieves 100% line coverage on the reference Verilog, and a formal equivalence check in which both the reference and the generated code are synthesized and compared. The testbench closes the gap left by earlier benchmarks' low coverage, and the formal check closes the remaining gap between functional correctness on sampled inputs and full equivalence. The paper also uses a two-level task split: module-level tasks provide submodule information and golden submodule implementations, while system-level tasks provide only the full hierarchical specification and require generating everything.

What would settle it

Have experienced Verilog engineers who have not seen the reference implementations attempt to implement a sample of RealBench modules from the design specifications alone; if a substantial fraction of such tasks also fail, the low LLM scores reflect specification quality rather than code-generation ability, and the benchmark's central comparison would need reinterpretation.

Watch

Extended reading notes

Core claim

The central claim is that real-world Verilog generation, at the level of complete IP modules and entire systems, remains largely unsolved by current LLMs, and that prior benchmarks hid this by using small flat designs, thin text-only specifications, and testbenches that do not fully exercise the reference code. RealBench counters this with 60 module-level and 4 system-level tasks drawn from an SD card controller, AES encryption cores, and a RISC-V CPU, each paired with a manually rewritten hierarchical specification containing text, diagrams, interface tables, register lists, corner cases, and constraints. Verification combines testbenches with 100% line coverage of the reference implementation and a formal equivalence check between synthesized netlists of the reference and the generated code. On this setup, the paper reports that o1-preview passes only 13.3% of module-level tasks on the first attempt and none of the system-level tasks; reasoning models lose their usual edge on complex tasks, multi-modal models gain little from diagrams, and modules requiring submodule instantiation or finite-state machines are the most frequent failures.

Load-bearing premise

The handwritten design specifications are complete, unambiguous, and faithful to the reference implementations, so that a skilled engineer could produce the correct Verilog from the specification alone; the paper states this as its design principle but does not test it with engineers who have not seen the reference code.

Editorial extensions

If this is right

  • Prior benchmark results overstate LLM capability: the paper reports that 44.2% of GPT-4-Turbo code that passes RTLLMV2 testbenches fails formal equivalence checking.
  • The best tested model reaches only 13.3% formal@1 on module-level RealBench tasks, and no tested model completes any system-level task.
  • Formal verification is necessary for reliable evaluation, since even 100% line-coverage testbenches leave a large gap between func@1 and formal@1 in the reported results.
  • Submodule instantiation is a major bottleneck: the paper reports that formal@1 is about 13 percentage points lower on average for tasks with submodules than without, and the best model's failures concentrate on non-leaf modules.
  • Reasoning models lose their advantage on complex tasks: DeepSeek-R1-671B improves over DeepSeek-V3-671B on prior benchmarks but shows roughly a 3 percentage point drop on RealBench formal@1, with hallucinated module headers cited as a cause.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A useful extension would be a scored partial-credit version of system-level tasks, since the current 0% pass rate treats a nearly working system identically to a syntax-error dump and gives no signal about which sub-systems are within reach.
  • The paper's finding that multi-modal models gain little from diagrams may depend on how diagrams are rendered; a controlled ablation that varies diagram clarity, resolution, and layout could separate model weakness from benchmark presentation.
  • If the specification-sufficiency premise holds, RealBench could serve as a training signal generator: failures on submodule instantiation suggest a direct next step of fine-tuning on hierarchical interface-aware code, and the formal checker could score such training data automatically.
  • The authors' protocol of 100% line-coverage testbenches plus formal equivalence could become a standard for future RTL benchmarks, but its cost on large designs will likely push the field toward cheaper approximate formal checks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. RealBench is a new benchmark for evaluating LLM-generated Verilog against real-world IP designs. It comprises 60 module-level tasks and 4 system-level tasks drawn from four open-source IPs (an SD card controller, AES cores, and the Hummingbirdv2 E203 CPU). For each task the authors manually write a multi-modal design specification (text, diagrams, tables), and correctness is judged by a 100% line-coverage testbench plus formal equivalence between the Yosys-synthesized netlists of the generated code and the reference implementation, checked with Cadence JasperGold. The paper evaluates ten LLMs and two agent frameworks, reporting low pass rates: the best model, o1-preview, achieves 13.3% formal@1 on module-level tasks and 0% on system-level tasks, with other models scoring lower. The authors also analyze failure modes related to submodule instantiation and FSM complexity, and compare their results with prior benchmarks (RTLLMV2, VerilogEvalV2, ChipGPTV).

Significance. If the benchmark is as described, it is a valuable contribution: it is, to my knowledge, the first IP-level Verilog generation benchmark with real-world open-source designs, multi-modal and structured specifications, and formal verification. The public release of the benchmark, including specifications, testbenches, and verification scripts, enables reproducible comparisons and will likely be useful to the hardware design automation community. The evaluation is extensive (ten models, two agents) and the observation that current LLMs perform far worse on real-world-style tasks than on existing benchmarks is an important and timely finding. The paper also provides a standardized decontamination protocol, which addresses a real gap in prior work. However, the benchmark's validity rests on two assumptions that are not fully substantiated: (i) the handwritten specifications are complete and unambiguous enough that a functionally correct design should match the reference implementation exactly, and (ii) the formal equivalence check is configured in a way that correctly distinguishes spec-conformant from non-conformant designs.

major comments (4)
  1. [II-C] The formal equivalence checking procedure is described too briefly to be reproducible or interpretable. The paper states only that both the reference and generated Verilog are synthesized with Yosys and compared with Cadence JasperGold, but it does not specify whether the check is combinational or sequential equivalence, how registers are mapped between the two netlists, whether initial states or reset values are constrained, or what time-frame bound is used. If the check requires exact register-level equivalence, then a functionally correct design that uses a different FSM state encoding, a different internal pipeline organization, or a different reset polarity would be rejected even if it fully satisfies the specification. The authors should provide the complete formal configuration (including scripts), report how many tasks fail equivalence but pass the testbench, and give a breakdown of the failure reasons. Without this information, the headline formal@1 numbers may measure distance to one particular golden netlist rather than the ability to implement the spec.
  2. [II-B] The claim that the handwritten design specifications are complete, unambiguous, and faithful to the reference implementations is asserted but not independently validated. Section II-B states as a design principle that another engineer can correctly implement the corresponding Verilog solely from the specification, but no human study or third-party evaluation is reported. Because the specifications were co-designed with the reference code and testbenches, they may contain implementation-specific details (e.g., particular FSM encodings, register names, or simulation delays) that a real-world specification would not impose, or they may omit constraints that are necessary to reproduce the exact reference behavior. If the spec does not uniquely determine the implementation, the formal equivalence criterion conflates functional correctness with implementation matching. The authors should validate the specs with engineers who are blind to the reference code, or at minimum systematically analyze the specs for over-specification, and they should report how the pass rates change under a more functional correctness criterion (e.g., testbench plus a set of formal properties rather than full equivalence).
  3. [III-B / Table III] The statistical support for the headline results is weak. The paper samples o1-preview only once per task, as acknowledged in Section III-B, and no confidence intervals are reported for any pass@1 or pass@5 estimate. With 60 module-level tasks, an observed 13.3% pass@1 corresponds to a 95% confidence interval of roughly 5–22%, which overlaps with DS-V3-671B's 10.8% result. The claim that o1-preview achieves 'only 13.3%' and the implied ordering of models should therefore be phrased with appropriate uncertainty. The authors should sample o1-preview multiple times (or at least report the variance from repeated sampling of the other models) and report confidence intervals for the key numbers. This is essential for a benchmark paper whose central message is comparative model capability.
  4. [III-B (Formal verification is necessary)] The paper uses the observed gap between func@1 and formal@1 to argue that formal verification is necessary for reliable evaluation. This argument is weakened by the possibility, raised in my first comment, that the formal equivalence check is overly strict. If the formal check rejects implementations that are functionally equivalent to the reference but differ in non-specified implementation details, then the gap reflects the strictness of the equivalence criterion rather than the insufficiency of the testbenches. To support the claim, the authors should validate the formal check by demonstrating that all formal failures correspond to genuine functional deviations, for example by running the failing generated designs against additional random test cases or by showing that the reference implementation passes the formal check with a non-trivial margin. Otherwise, the 'formal verification is necessary' conclusion is not fully established.
minor comments (6)
  1. [III-A] The model abbreviation list contains a typo: 'DS-R1-671B: The 671B DeepSeek-R3 model' should be 'DeepSeek-R1'.
  2. [Table II] The naming of designs and modules is inconsistent; for example, 'sdc controller' and 'sd controller wb' appear in a way that makes it difficult to map rows to the design hierarchy. Please use consistent capitalization and separators.
  3. [II-E] The decontamination threshold (RougeL > 0.5) is stated without justification; please report the distribution of RougeL scores in the candidate training data and show sensitivity to the threshold.
  4. [III-A] The exact prompt template and task input format for the LLMs are not given in the paper; if the benchmark is open-sourced, please include the prompt or a direct reference to the exact prompt file used in the evaluation to allow precise reproduction.
  5. [III-B] The phrase 'there remains a gap of 0% to 100% relatively between func@1 and formal@1' is unclear; presumably it means the relative gap ranges from 0% to 100% across models. Please rephrase.
  6. [References] Reference [42] is for DeepSeek-R1 but the title says 'DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning' - the model name in the text should be R1, not R3.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RealBench reports external pass-rate measurements against reference designs, with no fitted parameter or self-citation chain forcing the headline results.

full rationale

RealBench is a benchmark paper, not a derivation. Its central empirical claims are measured pass rates of LLMs on module-level and system-level Verilog generation tasks, evaluated against open-source reference IPs (SD card controller, AES cores, Hummingbirdv2 E203). These pass rates are external measurements: the reference Verilog comes from OpenCores and the E203 CPU, the testbenches and formal equivalence checks are applied to LLM outputs, and success is defined by matching the reference behavior, not by any quantity defined in terms of the paper's own equations. No parameter is fitted to a subset of data and then renamed as a prediction; no result is derived from an assumption that already contains the conclusion. The paper does cite prior work by overlapping authors, notably CodeV, but only as a baseline model evaluated on the benchmark; this self-citation is not load-bearing in the sense that the benchmark's correctness or the reported scores depend on CodeV's validity. The manually rewritten design specifications could affect benchmark validity, and the strictness of formal equivalence to a single golden implementation could understate capability, but these are concerns about whether the test measures what it claims, not circularity: the specification is not defined in terms of the pass rates, and the pass rates are not constructed to equal the benchmark's assumptions. The paper is self-contained against external reference designs and verification tools, so the most appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark contains no fitted physical parameters or invented theoretical entities. The central measurement rests on assumptions about toolchain fidelity, specification completeness, and representativeness of the selected IPs rather than on free parameters.

assumptions (4)
  • domain assumption Yosys synthesis plus Cadence JasperGold equivalence checking is a faithful oracle for functional correctness of generated Verilog.
    Section II-C uses this workflow to declare formal pass or fail; if synthesis or equivalence setup mishandles generated code, the pass rates are incorrect.
  • domain assumption The manually rewritten design specifications are complete, unambiguous, and faithful, so that a competent engineer could implement the reference solely from them.
    Section II-B states this as the design principle but provides no independent blind verification; the entire benchmark validity rests on this.
  • domain assumption The four selected open-source IPs, SD controller, AES cores, and E203 CPU, are representative of real-world IP design workflows.
    Section II-A selects designs for diversity but does not justify representativeness beyond these four examples.
  • domain assumption 100% line coverage of the reference code plus formal equivalence is sufficient to judge generated code correctness.
    Section II-C claims 100% line coverage makes verification more reliable; line coverage is a weak metric on its own, though formal checking complements it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs." pith.science (2026). https://pith.science/paper/EJLQE5FP

@misc{pith2026250716200,
  author       = {Pith},
  title        = {Pith review of: RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJLQE5FP}},
  note         = {Machine review of arXiv:2507.16200}
}
read the original abstract

The automatic generation of Verilog code using Large Language Models (LLMs) has garnered significant interest in hardware design automation. However, existing benchmarks for evaluating LLMs in Verilog generation fall short in replicating real-world design workflows due to their designs' simplicity, inadequate design specifications, and less rigorous verification environments. To address these limitations, we present RealBench, the first benchmark aiming at real-world IP-level Verilog generation tasks. RealBench features complex, structured, real-world open-source IP designs, multi-modal and formatted design specifications, and rigorous verification environments, including 100% line coverage testbenches and a formal checker. It supports both module-level and system-level tasks, enabling comprehensive assessments of LLM capabilities. Evaluations on various LLMs and agents reveal that even one of the best-performing LLMs, o1-preview, achieves only a 13.3% pass@1 on module-level tasks and 0% on system-level tasks, highlighting the need for stronger Verilog generation models in the future. The benchmark is open-sourced at https://github.com/IPRC-DIP/RealBench.

Figures

Figures reproduced from arXiv: 2507.16200 by the authors.

Figure 3
Figure 3. Compared with verification processes in other benchmarks, our [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Compared with design specifications in other benchmarks, our design [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Self-reflection agent overview. The agent generates Verilog code, [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Design hierarchy and o1-preview’s formal@1 of (a) SD card controller, (b) AES encoder/decoder core, and (c) Hummingbirdv2 E203 CPU Core. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Results of two LLM agent frameworks on VerilogEvalV2-spec2rtl and RealBench: (a) Self-reflection agent and (b) VerilogCoder. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The pass rate of o1-preview and GPT-4o-V varies with the number [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HAVEN: Hybrid Automated Verification ENgine for UVM Testbench Synthesis with LLMs

    cs.AR 2026-04 unverdicted novelty 7.0 of 10

    HAVEN combines LLM agents for planning and gap analysis with protocol-specific templates and a custom DSL to generate correct UVM testbenches, achieving 100% compilation success, 90.6% code coverage, and 87.9% functio...

  2. Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows

    cs.AI 2026-07 conditional novelty 6.0 of 10

    LLM agents can complete an RTL-to-GDS chip flow, but reliable completion depends on the execution infrastructure, not the foundation model alone.

  3. From Indiscriminate to Targeted: Efficient RTL Verification via Functionally Key Signal-Driven LLM Assertion Generation

    cs.AR 2026-04 unverdicted novelty 6.0 of 10

    AgileAssert identifies top critical signals via hybrid scoring on RTL graphs and uses structure-aware slicing to let LLMs generate targeted assertions, cutting assertion count by 66.68% and token use by 64% while matc...

  4. InCoder-32B-Thinking: Industrial Code World Model for Thinking

    cs.AR 2026-04 unverdicted novelty 6.0 of 10

    InCoder-32B-Thinking uses error-feedback synthesized thinking traces and a code world model to reach top open-source scores on general and industrial code benchmarks including 81.3% on LiveCodeBench and 84.0% on CAD-Coder.

  5. ChipVerilog: A Large-Scale OpenCores-Derived Benchmark for LLM-Based Verilog RTL Generation

    cs.AR 2026-07 conditional novelty 5.0 of 10

    On 64 large OpenCores-derived Verilog tasks, top LLMs reach 23.6% functional pass@1, 37.5% pass@5, and 0% on designs with two or more submodules, showing hierarchical RTL generation remains unsolved.

  6. Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development

    cs.SE 2026-05 unverdicted novelty 4.0 of 10

    Agentic Agile-V uses Agile-V as backbone and a Specify-Constrain-Orchestrate-Prove-Evolve-Verify loop to convert AI agent conversations into traceable engineering artifacts with acceptance evidence.

Reference graph

Works this paper leans on

47 extracted references · 20 canonical work pages · cited by 6 Pith papers

  1. [1]

    For better or worse, benchmarks shape a field,

    D. Patterson, “For better or worse, benchmarks shape a field,” Commu- nications of the ACM , vol. 55, 2012

    work page 2012

  2. [2]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  3. [3]

    Chipnemo: Domain- adapted llms for chip design,

    M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu et al. , “Chipnemo: Domain- adapted llms for chip design,” arXiv preprint arXiv:2311.00176 , 2023

    arXiv 2023

  4. [4]

    Rtlcoder: Outper- forming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution,

    S. Liu, W. Fang, Y . Lu, Q. Zhang, H. Zhang et al., “Rtlcoder: Outper- forming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution,” arXiv preprint arXiv:2312.08617 , 2023

    arXiv 2023

  5. [5]

    Betterv: Con- trolled verilog generation with discriminative guidance,

    Z. Pei, H.-L. Zhen, M. Yuan, Y . Huang, and B. Yu, “Betterv: Con- trolled verilog generation with discriminative guidance,” arXiv preprint arXiv:2402.03375, 2024

    arXiv 2024

  6. [6]

    Mg-verilog: Multi-grained dataset towards enhanced llm-assisted verilog generation,

    Y . Zhang, Z. Yu, Y . Fu, C. Wan et al. , “Mg-verilog: Multi-grained dataset towards enhanced llm-assisted verilog generation,”arXiv preprint arXiv:2407.01910, 2024

    arXiv 2024

  7. [7]

    Codev: Empowering llms for verilog generation through multi-level summarization,

    Y . Zhao, D. Huang, C. Li, P. Jin, Z. Nan et al. , “Codev: Empowering llms for verilog generation through multi-level summarization,” arXiv preprint arXiv:2407.10424, 2024

    arXiv 2024

  8. [8]

    Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework

    K. Chang, K. Wang, N. Yang, Y . Wang, D. Jin, W. Zhu, Z. Chen, C. Li, H. Yan, Y . Zhou et al., “Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework,” arXiv preprint arXiv:2403.11202, 2024

    work page Pith review arXiv 2024

Show all 47 references
  1. [9]

    Autovcoder: A systematic framework for automated verilog code generation using llms,

    M. Gao, J. Zhao, Z. Lin, W. Ding, X. Hou, Y . Feng, C. Li, and M. Guo, “Autovcoder: A systematic framework for automated verilog code generation using llms,” arXiv preprint arXiv:2407.18333 , 2024

    2024 arXiv

  2. [10]

    Craftrtl: High-quality synthetic data generation for verilog code models with correct-by- construction non-textual representations and targeted code repair,

    M. Liu, Y .-D. Tsai, W. Zhou, and H. Ren, “Craftrtl: High-quality synthetic data generation for verilog code models with correct-by- construction non-textual representations and targeted code repair,” arXiv preprint arXiv:2409.12993, 2024

    2024 arXiv

  3. [11]

    Aivril: Ai-driven rtl gen- eration with verification in-the-loop,

    H. Sami, P.-E. Gaillardon, V . Tenace et al. , “Aivril: Ai-driven rtl gen- eration with verification in-the-loop,” arXiv preprint arXiv:2409.11411 , 2024

    2024 arXiv

  4. [12]

    Chipgpt: How far are we from natural language hardware design,

    K. Chang, Y . Wang, H. Ren, M. Wang, S. Liang et al., “Chipgpt: How far are we from natural language hardware design,” 2023

    2023

  5. [13]

    Chip-chat: Chal- lenges and opportunities in conversational hardware design,

    J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-chat: Chal- lenges and opportunities in conversational hardware design,” in 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD) . IEEE, sep 2023

    2023

  6. [14]

    Rtlrewriter: Methodologies for large models aided rtl code optimization,

    X. Yao, Y . Wang, X. Li, Y . Lian, R. Chen, L. Chen, M. Yuan, H. Xu, and B. Yu, “Rtlrewriter: Methodologies for large models aided rtl code optimization,” arXiv preprint arXiv:2409.11414 , 2024

    2024 arXiv

  7. [15]

    The dawn of ai-native eda: Promises and challenges of large circuit models,

    L. Chen, Y . Chen, Z. Chu, W. Fang, T.-Y . Ho et al. , “The dawn of ai-native eda: Promises and challenges of large circuit models,” arXiv preprint arXiv:2403.07257, 2024

    2024 arXiv

  8. [16]

    Assertllm: Generating and evaluating hardware verification assertions from design specifications via multi-llms,

    Z. Yan, W. Fang, M. Li, M. Li, S. Liu, Z. Xie, and H. Zhang, “Assertllm: Generating and evaluating hardware verification assertions from design specifications via multi-llms,” arXiv preprint arXiv:2402.00386 , 2024

    2024

  9. [17]

    Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models,

    Y . Fu, Y . Zhang, Z. Yu, S. Li, Z. Ye, C. Li, C. Wan, and Y . C. Lin, “Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models,” in 2023 IEEE/ACM International Confer- ence on Computer Aided Design (ICCAD) . IEEE, 2023, pp. 1–9

    2023

  10. [18]

    Chateda: A large language model powered autonomous agent for eda,

    H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng et al. , “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2024

    2024

  11. [19]

    Llm-assisted generation of hardware assertions,

    R. Kande, H. Pearce, B. Tan, B. Dolan-Gavitt, S. Thakur, R. Karri, and J. Rajendran, “Llm-assisted generation of hardware assertions,” arXiv preprint arXiv:2306.14027, 2023

    2023 arXiv

  12. [20]

    Rtlfixer: Automatically fixing rtl syntax errors with large language model,

    Y . Tsai, M. Liu, and H. Ren, “Rtlfixer: Automatically fixing rtl syntax errors with large language model,” in Proceedings of the 61st ACM/IEEE Design Automation Conference , 2024, pp. 1–6

    2024

  13. [21]

    Hdldebugger: Streamlining hdl debugging with large language models,

    X. Yao, H. Li, T. H. Chan, W. Xiao, M. Yuan et al. , “Hdldebugger: Streamlining hdl debugging with large language models,” arXiv preprint arXiv:2403.11671, 2024

    2024 arXiv

  14. [22]

    On the viability of using llms for sw/hw co-design: An example in designing cim dnn accelerators,

    Z. Yan, Y . Qin, X. S. Hu, and Y . Shi, “On the viability of using llms for sw/hw co-design: An example in designing cim dnn accelerators,” in 2023 IEEE 36th International System-on-Chip Conference (SOCC) . IEEE, 2023, pp. 1–6

    2023

  15. [23]

    Openllm-rtl: Open dataset and benchmark for llm-aided design rtl generation,

    S. Liu, Y . Lu, W. Fang, M. Li, and Z. Xie, “Openllm-rtl: Open dataset and benchmark for llm-aided design rtl generation,” 2024

    2024

  16. [24]

    Benchmarking large language models for auto- mated verilog rtl code generation,

    S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan- Gavitt, and S. Garg, “Benchmarking large language models for auto- mated verilog rtl code generation,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE) . IEEE, 2023, pp. 1–6

    2023

  17. [25]

    Verilogeval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) . IEEE, 2023, pp. 1–8

    2023

  18. [26]

    Rtllm: An open-source benchmark for design rtl generation with large language model,

    Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) . IEEE, 2024, pp. 722–727

    2024

  19. [27]

    Natural language is not enough: Benchmarking multi-modal generative ai for verilog generation,

    K. Chang, Z. Chen, Y . Zhou, W. Zhu, H. Xu et al., “Natural language is not enough: Benchmarking multi-modal generative ai for verilog generation,” arXiv preprint arXiv:2407.08473 , 2024

    2024 arXiv

  20. [28]

    Revisiting verilogeval: Newer llms, in-context learning, and specification-to-rtl tasks,

    N. Pinckney, C. Batten, M. Liu, H. Ren, and B. Khailany, “Revisiting verilogeval: Newer llms, in-context learning, and specification-to-rtl tasks,” arXiv preprint arXiv:2408.11053 , 2024

    2024 arXiv

  21. [29]

    Rtl-repo: A benchmark for evaluating llms on large-scale rtl design projects,

    A. Allam and M. Shalan, “Rtl-repo: A benchmark for evaluating llms on large-scale rtl design projects,” arXiv preprint arXiv:2405.17378 , 2024

    2024 arXiv

  22. [30]

    Specllm: Exploring generation and review of vlsi design specification with large language model,

    M. Li, W. Fang, Q. Zhang, and Z. Xie, “Specllm: Exploring generation and review of vlsi design specification with large language model,”arXiv preprint arXiv:2401.13266, 2024

    2024 arXiv

  23. [31]

    Opencores: Open source ip-cores,

    OpenCores, “Opencores: Open source ip-cores,” https://opencores.org/

  24. [32]

    Design sdc mmc controller,

    ——, “Design sdc mmc controller,” https://github.com/freecores/sdcard mass storage controller

  25. [33]

    Hummingbirdv2 e203 core and soc,

    riscv mcu, “Hummingbirdv2 e203 core and soc,” https://github.com/riscv-mcu/e203 hbirdv2/tree/master

  26. [34]

    SWE-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real- world github issues?” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https: //openreview.net/forum...

    2024

  27. [35]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,” Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  28. [36]

    Yosys open synthesis suite,

    C. Wolf, “Yosys open synthesis suite,” https://yosyshq.net/yosys/

  29. [37]

    Nangate freepdk45 generic open cell library,

    Silvaco, “Nangate freepdk45 generic open cell library,” https://si2.org/open-cell-library/

  30. [38]

    Jasper formal verification platform,

    Cadence, “Jasper formal verification platform,” https://www.cadence. com/en US/home/tools/system-design-and-verification/formal-and- static-verification/jasper-verification-platform.html

  31. [39]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  32. [40]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  33. [41]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437 , 2024

    2024 arXiv

  34. [42]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang et al. , “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  35. [43]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong et al. , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196 , 2024

    2024 arXiv

  36. [44]

    Code with codeqwen1.5,

    Q. Team, “Code with codeqwen1.5,” April 2024. [Online]. Available: https://qwenlm.github.io/blog/codeqwen1.5/

    2024

  37. [45]

    Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)- based waveform tracing tool,

    C.-T. Ho, H. Ren, and B. Khailany, “Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)- based waveform tracing tool,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 39, no. 1, 2025, pp. 300–307

    2025

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  39. [47]

    Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,

    F. Cui, C. Yin, K. Zhou, Y . Xiao, G. Sun et al. , “Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,” in Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, 2024, pp. 1–9

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.