Pith. sign in

REVIEW 4 major objections 5 minor 18 references

EDA-Aware RTL Generation with Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A self-verifying agentic framework lifts LLM-generated RTL functional pass rates to 77% for Verilog and 66% for VHDL.

desk verdict The Verilog core result is worth peer review; the VHDL headline is not yet supported because the testbench provenance is missing. read the letter →

arxiv 2412.04485 v1 pith:CCVHHSWS submitted 2024-11-21 cs.AR cs.AI

Mubashir ul Islam , Humza Sami , Pierre-Emmanuel Gaillardon , Valerio Tenace This is my paper · ORCID
classification cs.ARcs.AI
keywords RTLcodegenerationLargelanguagemodelsVerilogVHDLMulti-agentsystemsSelf-verificationEDAfeedbackFunctionalverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the main bottleneck in LLM-generated hardware description code is not the model's raw ability but the absence of automated correction loops, and that a language-agnostic agentic framework can close most of the gap. Its proposed system, AIVRIL2, separates correction into a syntax loop and a functional loop: a Code Agent writes both the testbench and the RTL, a Review Agent converts Vivado compiler logs into corrective prompts, and a Verification Agent converts simulator failure logs into behavioral corrective prompts while the testbench is held fixed. On all 156 VerilogEval-Human problems, the best configuration (Claude 3.5 Sonnet) reaches 100% syntax $\mathrm{pass@1}$ and 77% functional $\mathrm{pass@1}$ for Verilog and 66% functional $\mathrm{pass@1}$ for VHDL, which the paper reports as a 3.4x improvement over the best prior method and a large lift over each model's zero-shot baseline. A sympathetic reader would care because it suggests generic LLMs, without fine-tuning or retrieval augmentation, can be steered toward hardware-grade correctness by closing the loop with EDA tool feedback.

What carries the argument

The key machinery is a two-stage, testbench-first feedback loop. A testbench is first generated from the user prompt and frozen as the oracle; a Review Agent reads Vivado compiler logs and emits corrective syntax prompts, while a Verification Agent reads simulator logs and emits corrective functional prompts; both feed a single Code Agent that regenerates the RTL. What does the work is the invariant that the testbench never changes across iterations, so each functional pass/fail measurement is a comparison of the new RTL against the same unchanging criterion, and the error logs are distilled into targeted prompts rather than raw text.

What would settle it

Rerun the Claude 3.5 Sonnet configuration on the same 156 problems with the same hyperparameters but using an independently created, human-checked VHDL testbench suite in place of the paper's unspecified VHDL testbenches; if the functional pass rate falls well below 66%, the VHDL claim depends on the missing testbench provenance rather than on the framework.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a fixed testbench generated from the user specification, combined with two alternating correction loops driven by real EDA tool logs, turns weak zero-shot RTL generations into designs that pass functional checks. The Syntax Optimization loop feeds compiler diagnostics through a Review Agent that returns line-level corrections, and the Functional Optimization loop feeds simulator traces through a Verification Agent that returns behavior-level corrections; only the RTL changes between iterations. With Claude 3.5 Sonnet this lifts Verilog $\mathrm{pass@1S}$ from 91.03% to 100% and $\mathrm{pass@1F}$ from 60.23% to 77%, and VHDL $\mathrm{pass@1S}$ from 88.46% to 100% and $\mathrm{pass@1F}$ from 53.85% to 66%. The same machinery works for weaker models: Llama3-70B goes from 1.28% syntax $\mathrm{pass@1}$ for VHDL to 58.87%, and from 0% functional $\mathrm{pass@1}$ to 32.69%.

Load-bearing premise

The 66% VHDL functional pass rate assumes that valid, independent VHDL testbenches exist and are used for all 156 VerilogEval-Human design problems, but the paper never says where those VHDL testbenches come from or how they were checked.

Editorial extensions

If this is right

  • Functional $\mathrm{pass@1}$ of 77% for Verilog and 66% for VHDL is reachable from a generic commercial LLM with no fine-tuning and no retrieval augmentation.
  • Syntax errors can be driven to zero in all reported configurations except Llama3-70B on VHDL, which still improves from a 1.28% baseline to 58.87%.
  • Even an LLM with essentially no VHDL training data (Llama3-70B, 0% functional baseline) can be recovered to 32.69% functional $\mathrm{pass@1}$ by the same loops.
  • The average functional improvement over corresponding baselines is at least 38.28 percentage points for Verilog and 69.44 percentage points for VHDL.
  • The framework's language-agnostic design implies the same two-loop correction structure should transfer to other HDLs, provided matching EDA tool logs are available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the oracle testbench is written by the same Code Agent that writes the RTL, the reported functional pass rates measure self-consistency against an LLM-authored specification of 'correct' at least as much as genuine conformance to the original natural-language intent.
  • Editorial inference: the 66% VHDL figure rests on testbenches whose provenance is not described in the paper; a plausible stress test is to rerun the VHDL loop with independently written, human-verified testbenches and compare functional $\mathrm{pass@1}$.
  • Editorial inference: the fixed-testbench invariant could be pushed further by perturbing the testbench between iterations (mutation or coverage-guided expansion), which may catch overfitting to the initial oracle and improve generalization to unseen test cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript presents AIVRIL2, a multi-agent LLM framework for RTL code generation that combines a syntax optimization loop and a functional optimization loop. The loops use a Code Agent, a Review Agent, and a Verification Agent, with corrective feedback derived from EDA tool logs (Vivado). The framework is evaluated on all 156 VerilogEval-Human problems for both Verilog and VHDL, using Claude 3.5 Sonnet, GPT-4o, and Llama3-70B. Reported results include 100% syntax pass rates for five of six configurations and functional pass rates of 77% (Verilog) and 66% (VHDL) for Claude 3.5 Sonnet, stated as a 3.4x improvement over prior methods.

Significance. If the Verilog results are reproducible, this is a useful practical contribution: it demonstrates that iterative EDA-log feedback can raise zero-shot LLM syntax pass rates to near 100% and functional pass rates substantially, and it is one of the first attempts to evaluate LLM-generated VHDL on the VerilogEval-Human problem set. The framework is LLM-agnostic in principle, uses standard EDA tools, and the Verilog final metric appears to rely on the official benchmark testbenches. However, the strength of the empirical claims is limited by the missing provenance of the VHDL evaluation artifacts, the selective comparison used for the 3.4x headline, and the absence of any variance or reproducibility analysis.

major comments (4)
  1. [§4.1, Table 1] Section 4.1 states that pass@1F was “determined by executing the testbenches provided in the benchmark suite,” but VerilogEval-Human is a Verilog-only benchmark, so there are no VHDL testbenches “provided in the benchmark suite.” The manuscript does not describe how the 156 problems were translated to VHDL, who wrote the VHDL testbenches, or how those testbenches were validated. This matters because Section 3.1 makes the Code Agent generate the testbench before RTL generation; if the VHDL testbenches were produced in that same way, the 66% pass@1F for Claude 3.5 Sonnet in Table 1 would measure self-consistency rather than functional correctness. The authors need to provide the VHDL benchmark and testbench artifacts and specify their provenance before the VHDL claims can be accepted.
  2. [Abstract, §4.3, Table 2] The abstract’s “3.4x enhancement over prior methods” is based on comparing the 77% Verilog pass@1F with ChipNemo-13B’s 22.4% in Table 2, which is the lowest value in that table. The closest prior system, AIVRIL, reaches 67.3%, so the improvement over the strongest listed prior approach is about 1.14x, not 3.4x. As written, the headline overstates the advance and should be reworded to say “3.4x over the weakest listed baseline” or should compare with the best prior result.
  3. [§4.1, Table 1] No repeated runs, confidence intervals, or variance information are reported. The unbiased pass@k estimator with k=1 is simply the empirical pass rate; given stochastic LLM sampling at temperature 0.2 and top_p 0.1, the reported differences (e.g., 72.44 vs. 77 for GPT-4o and Claude 3.5 Sonnet) may not be stable. The paper should report multiple seeds or at least bootstrap intervals for the headline pass@1F numbers.
  4. [§3.1–§3.3] The functional optimization loop uses a self-generated testbench as its internal oracle, and the stopping condition is that testbench’s “All tests passed successfully!” message. Even if the final Verilog metric uses the official benchmark testbenches, the paper provides no evidence that a self-generated testbench pass implies a pass on the official testbench (for example, an analysis of how often the internal loop stopped on a self-generated pass but failed the official testbench). Without such an analysis, the framework’s internal self-verification claim is incomplete.
minor comments (5)
  1. [Throughout] The framework name is written inconsistently as “AIVRIL 2” and “AI VRIL 2”; please standardize.
  2. [§4.1] The reference to the unbiased pass@k estimator [14] is confusing for k=1, where it reduces to an empirical proportion; consider simply reporting the fraction of passing designs.
  3. [Table 1] The “N/A” in the VHDL ΔF cell for the AIVRIL2 (Llama3-70B) row is unexplained; since the baseline is 0%, the caption should define how an improvement is computed in that case.
  4. [Figure 3] The caption says the latency “accounts for the execution times of EDA tools,” but the bars appear to include LLM response time; clarify whether LLM API call time is included and report error bars or per-iteration distributions.
  5. [§4.2] The reported average improvements of 38.28% for Verilog and “at least 69.44%” for VHDL are not derived in the text; showing the averaging formula would help reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

VHDL pass@1F rests on a self-generated testbench oracle; Verilog results are externally grounded.

  1. self definitional [Section 3.1 (Code Agent workflow), Section 4.1 (Methodology), Table 1 (VHDL column)]
    "the agent first generates a comprehensive testbench based on the received specifications, ensuring that all potential test cases, that a functionally correct RTL design must pass, are covered. | I want you to generate RTL that passes all the checks in the provided testbench. | Notably, pass@kF was determined by executing the testbenches provided in the benchmark suite, ensuring a comprehensive validation of the overall approach."

    For VHDL this reduces to self-consistency: VerilogEval-Human is a Verilog-only benchmark, and the paper itself says in Section 4.3 that this is the first evaluation of GenAI solutions for VHDL, so no VHDL testbenches are provided in the benchmark suite. The only VHDL testbenches described in the pipeline are generated by the Code Agent from the same user specification that is later used to generate the RTL, and the RTL prompt explicitly asks for code that passes that generated testbench. Thus the reported VHDL pass@1F measures whether the model's RTL satisfies a testbench written by the same model from the same prompt; it is an internal consistency check, not an independent functional oracle.

full rationale

The Verilog evaluation chain is not circular: Section 4.1 states that pass@1F was determined by executing the benchmark-suite testbenches, and the paper uses the official VerilogEval-Human problems with their external testbenches; syntax errors are checked with Vivado logs; the final 77% Verilog pass@1F is therefore grounded in an independent oracle. The framework's internal self-generated testbench is explicitly presented as a self-verification mechanism, not as the final ground truth for the Verilog claim. The VHDL claim, however, is not equivalently grounded: VerilogEval-Human contains no VHDL testbenches, and the only VHDL testbench source described is the Code Agent, which generates the testbench before writing the RTL and keeps it fixed during functional optimization. Unless an independent, validated VHDL testbench suite was constructed and used, which the manuscript does not describe, the 66% VHDL figure and the language-agnostic claim are consistent with the model passing its own testbench rather than an external functional specification. No load-bearing self-citation chain was found: AIVRIL [7] is cited as prior work and appears in Table 2 as a baseline, but the central Verilog result does not depend on that citation. The 3.4x comparison is selective relative to ChipNemo-13B rather than the closer AIVRIL result, but that is a comparison-construction concern, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The ledger is dominated by evaluation assumptions rather than mathematical parameters. The framework itself introduces no new physical entities and no formal free parameters, but the reported pass rates depend on sampling hyperparameters, an undocumented iteration budget, and an unverified VHDL testbench translation.

free parameters (3)
  • LLM sampling temperature = 0.2
    Chosen by authors; no sensitivity analysis is reported, and temperature affects pass@1 outcomes in code generation.
  • top_p = 0.1
    Chosen alongside temperature; not swept, but can materially change generated RTL diversity and correctness.
  • per-loop iteration budget = not reported
    Section 3.3 says the loop stops at a predefined maximum number of iterations, but the actual cap is never stated. This budget trades off pass rate and latency.
assumptions (3)
  • domain assumption VerilogEval-Human testbenches are a valid and complete oracle for functional correctness of generated RTL.
    The entire pass@1F metric rests on the assumption that passing the provided benchmark testbench means the RTL is functionally correct. Invoked in Section 4.1.
  • domain assumption The self-generated testbench is informative for steering corrections without causing the RTL to overfit to spurious self-testbench behavior.
    The framework uses its own first-stage testbench for all functional feedback (Section 3.3). If this testbench is weak or biased, the corrective prompts may not improve true correctness.
  • ad hoc to paper The VHDL versions of the VerilogEval-Human problems preserve the original functional requirements.
    No translation or validation process for VHDL testbenches is described (Section 4.1), yet the 66% VHDL pass@1F depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EDA-Aware RTL Generation with Large Language Models." pith.science (2026). https://pith.science/paper/CCVHHSWS

@misc{pith2026241204485,
  author       = {Pith},
  title        = {Pith review of: EDA-Aware RTL Generation with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCVHHSWS}},
  note         = {Machine review of arXiv:2412.04485}
}
abstract

Large Language Models (LLMs) have become increasingly popular for generating RTL code. However, producing error-free RTL code in a zero-shot setting remains highly challenging for even state-of-the-art LLMs, often leading to issues that require manual, iterative refinement. This additional debugging process can dramatically increase the verification workload, underscoring the need for robust, automated correction mechanisms to ensure code correctness from the start. In this work, we introduce AIvril2, a self-verifying, LLM-agnostic agentic framework aimed at enhancing RTL code generation through iterative corrections of both syntax and functional errors. Our approach leverages a collaborative multi-agent system that incorporates feedback from error logs generated by EDA tools to automatically identify and resolve design flaws. Experimental results, conducted on the VerilogEval-Human benchmark suite, demonstrate that our framework significantly improves code quality, achieving nearly a 3.4$\times$ enhancement over prior methods. In the best-case scenario, functional pass rates of 77% for Verilog and 66% for VHDL were obtained, thus substantially improving the reliability of LLM-driven RTL code generation.

Figures

Figures reproduced from arXiv: 2412.04485 by the authors.

Figure 1
Figure 1. Architecture overview of the proposed AI [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Practical example of the proposed workflow and internal state representation of the agents in AI [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average latency breakdown across optimization loops for the proposed framework. Reported figures account [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 5 canonical work pages

  1. [1]

    Chipnemo: Domain-adapted llms for chip design,

    M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu et al., “Chipnemo: Domain-adapted llms for chip design,” arXiv preprint arXiv:2311.00176, 2023

    arXiv 2023

  2. [2]

    Chateda: A large language model powered autonomous agent for eda,

    H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024

    2024

  3. [3]

    Rtlfixer: Automatically fixing rtl syntax errors with large language models,

    Y . Tsai, M. Liu, and H. Ren, “Rtlfixer: Automatically fixing rtl syntax errors with large language models,”arXiv preprint arXiv:2311.16543, 2023

    arXiv 2023

  4. [4]

    Towards llm-powered verilog rtl assistant: Self- verification and self-correction,

    H. Huang, Z. Lin, Z. Wang, X. Chen, K. Ding, and J. Zhao, “Towards llm-powered verilog rtl assistant: Self- verification and self-correction,” arXiv preprint arXiv:2406.00115, 2024

    arXiv 2024

  5. [5]

    VerilogEval: evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2023

    work page 2023

  6. [6]

    Codev: Empowering llms for verilog generation through multi-level summarization,

    Y . Zhao, D. Huang, C. Li, P. Jin, Z. Nan, T. Ma, L. Qi, Y . Pan, Z. Zhang, R. Zhanget al., “Codev: Empowering llms for verilog generation through multi-level summarization,”arXiv preprint arXiv:2407.10424, 2024

    arXiv 2024

  7. [7]

    Aivril: Ai-driven rtl generation with verification in-the-loop,

    H. Sami, P.-E. Gaillardon, V . Tenaceet al., “Aivril: Ai-driven rtl generation with verification in-the-loop,”arXiv preprint arXiv:2409.11411, 2024

    arXiv 2024

  8. [8]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    work page 2022

Show all 18 references
  1. [9]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in International Conference on Machine Learning. PMLR, 2022, pp. 9118– 9147

    2022

  2. [10]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  3. [11]

    Chip-chat: Challenges and opportunities in conversational hardware design,

    J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-chat: Challenges and opportunities in conversational hardware design,” in 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD). IEEE, 2023, pp. 1–6

    2023

  4. [12]

    Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework,

    K. Chang, K. Wang, N. Yang, Y . Wang, D. Jin, W. Zhu, Z. Chen, C. Li, H. Yan, Y . Zhouet al., “Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework,” arXiv preprint arXiv:2403.11202, 2024

    2024 arXiv

  5. [13]

    Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool,

    C.-T. Ho, H. Ren, and B. Khailany, “Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool,” arXiv preprint arXiv:2408.08927, 2024

    2024 arXiv

  6. [14]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  7. [15]

    Claude 3.5 Sonnet Model Card Addendum,

    Anthropic, “Claude 3.5 Sonnet Model Card Addendum,” 2024

    2024

  8. [16]

    GPT-4o System Card,

    OpenAI, “GPT-4o System Card,” 2024

    2024

  9. [17]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Benchmarking large language models for automated verilog rtl code generation,

    S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan-Gavitt, and S. Garg, “Benchmarking large language models for automated verilog rtl code generation,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2023, pp. 1–6. 8

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.