Pith. sign in

REVIEW 4 major objections 6 minor 36 references

VRank: Enhancing Verilog Code Generation from Large Language Models via Self-Consistency

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VRank sorts Verilog candidates by testbench-output agreement and, with a chain-of-thought tie-break, raises pass@1 by 10.5% on average across four LLMs.

desk verdict A practical, fully-automatic selection method for LLM-generated Verilog with likely real gains, but the headline 10.5% needs error bars and cleaner reporting. read the letter →

arxiv 2502.00028 v1 pith:ELEPNCSN submitted 2025-01-22 cs.AR cs.PL

Zhuorui Zhao , Ruidi Qiu , Ing-Chao Lin , Grace Li Zhang , Bing Li , Ulf Schlichtmann This is my paper · ORCID
classification cs.ARcs.PL
keywords Verilogcodegenerationself-consistencylargelanguagemodelshardwaredesignautomationpass@1LLM-generatedtestbencheschain-of-thoughtMinimumBayesRisk
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VRank tackles the selection problem that appears whenever an LLM samples many Verilog modules from one specification: which candidate should be used? The paper proposes an automatic answer: run every candidate against a testbench that is itself generated by the LLM, group candidates whose simulation outputs are identical, and rank those groups by how many members they contain, on the idea that agreement tracks correctness. A chain-of-thought step then examines the test cases where the top two groups disagree, asks the LLM to predict the expected output, and swaps the ranking when repeated reasoning agrees with the second group. On the VerilogEval-Human benchmark this raises pass@1 by 6.7 to 15.7 percentage points across GPT-4o-mini, GPT-4o, Llama-3-70B, and CodeV-Qwen, for an average increase of 10.5%. If the claim holds, hardware designers can improve LLM-written RTL without writing golden testbenches or manually reviewing candidates.

What carries the argument

The load-bearing mechanism is execution-based clustering over an LLM-generated testbench. The LLM writes a set of test cases that print inputs and outputs without reference signals; every candidate module is simulated on those test cases; candidates with identical outputs on all test cases form a functional-equivalence cluster. Cluster score follows a Minimum Bayes Risk rule, $R(c)=n-\sum_{c'\in C}\ell_{\mathrm{strict}}(c,c')$, where $\ell_{\mathrm{strict}}(c,c')$ is 1 if any test-case output differs and 0 otherwise. For the top clusters, the framework finds a disagreeing test case, prompts the LLM zero-shot with chain-of-thought to predict the reference output, repeats this several times, and swaps the top two clusters if a supermajority (80%) of predictions agrees with the second cluster. This converts an unlabeled candidate pool into a ranked list using only LLM-generated signals.

What would settle it

Run VRank on VerilogEval-Human with the LLM-generated testbench replaced by a single non-discriminating test case that every candidate passes; if the top-cluster pass@1 does not collapse toward the random baseline, the reported gain is not being caused by the consistency signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that self-consistency of simulation outputs on an LLM-generated testbench is a reliable and fully automatic selection signal for Verilog code. In the paper's data for GPT-4o, 56.7% of the 7,800 candidates were correct and had at least one output-consistent partner, while only 14.6% were wrong yet consistent; almost all isolated candidates were wrong. Ranking clusters by a strict-consistency score and then applying chain-of-thought reasoning to resolve disagreements between the top clusters improved pass@1 on VerilogEval-Human for all four models tested, with the largest gain of 15.7 percentage points on CodeV-Qwen. The paper also reports that the gain is stable down to five samples, that roughly 90% of the 50-sample gain is reached at twenty samples, and that strict consistency beats case-wise consistency scoring.

Load-bearing premise

The framework rests on the assumption that output agreement on an LLM-written testbench separates correct from incorrect Verilog; the paper's own Table I shows the signal is noisy, since 1,135 of 5,561 consistent candidates (20.4%) were incorrect and 6 of the 2,239 inconsistent candidates were correct.

Editorial extensions

If this is right

  • An engineer can select a single Verilog module from a sampled pool without writing a human testbench or manually inspecting candidates, removing a major bottleneck in LLM-based RTL generation.
  • Because about 90% of the 50-sample gain is already reached at 20 candidates, users of expensive models can cut the sample count without giving up most of the benefit.
  • The selector is model-agnostic: it improved pass@1 for closed- and open-source models of different sizes, with the largest gain of 15.7 points going to the weakest baseline model.
  • Chain-of-thought tie-breaking adds a further increment beyond cluster ranking for GPT-4o and GPT-4o-mini, with the largest effect on problems described as application or connection specifications.
  • Strict all-or-nothing output agreement is the better consistency metric: the case-wise loss variant trails by 0 to 2 percentage points across models and sample sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the ceiling of the method is set by the LLM-generated testbench: if the testbench is not discriminating, consistent clusters of wrong answers will rank high, so a cheap testbench-quality check (for example, verifying that at least one candidate satisfies assertions on the specification) could harden the pipeline.
  • The same selection rule should transfer to other hardware description languages and to firmware or assembly generation, wherever a simulator can evaluate output equivalence for many sampled candidates.
  • The saturation near 20 samples suggests a compute-accuracy operating point: sampling more than about 20 candidates per module buys little, so the practical cost of VRank is dominated by simulation runs and chain-of-thought calls, not by candidate count.
  • A testable extension is to expose the second-ranked cluster as a disagreement alarm for human review, since the paper shows that for GPT-4o an engineer would only need to look at the top two clusters to find nearly all correct solutions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VRank, a fully automatic framework for improving pass@1 accuracy of LLM-generated Verilog code. VRank generates multiple code candidates and an LLM-generated testbench, clusters candidates by identical simulation outputs, ranks clusters by an MBR-style consistency score, and applies Chain-of-Thought reasoning to arbitrate between the top-ranked clusters. Experiments on VerilogEval-Human across GPT-4o-mini, GPT-4o, Llama-3-70B, and CodeV-Qwen report an average 10.5% increase in pass@1 relative to random selection, with an ablation on sample size and a comparison of strict versus case-wise loss functions.

Significance. If the reported gains are robust, VRank would offer a useful, human-free selection mechanism for hardware description generation, extending self-consistency ideas from NLP to RTL code. The paper tests on a public benchmark with 156 hand-crafted problems across four LLMs of different types, and the candidate-level analysis (7,800 samples for GPT-4o) is a strength. The main empirical claim, however, currently lacks statistical support, and one of the reported aggregates mixes evaluation conditions, so the size and reliability of the improvement are not yet established.

major comments (4)
  1. [Section V-B, Table III, Abstract] The headline claim of a "significant 10.5% average increase" is not supported by any measure of variability. The text in Section IV states that the experiment is repeated 5 times, but Table III reports only point estimates, with no standard deviations, confidence intervals, or hypothesis tests. Since the underlying consistency signal is noisy (Table I shows 20.4% of consistent candidates are incorrect), the reported gains could be within run-to-run variance. Please report mean ± std (or bootstrap CIs) across the repetitions for the key pass@1 numbers and test whether the improvement is statistically distinguishable from zero.
  2. [Section V-B, Table III] The "Increase" column mixes different evaluation protocols. For GPT-4o-mini and GPT-4o, the increase equals (CoT Reference Pass@1 - Baseline Pass@1), e.g., 58.1 - 48.8 = 9.3 and 67.8 - 57.4 = 10.4, respectively. For Llama-3 and CodeV-Qwen, it equals (non-CoT Our Method Pass@1 - Baseline Pass@1), e.g., 48.5 - 41.8 = 6.7 and 48.1 - 32.4 = 15.7. The table caption does not disclose this, and the "CoT Reference" column is undefined in the caption. This conflation makes the stated 10.5% average not a uniform comparison. Recompute the gains under a single protocol, or clearly separate CoT and non-CoT results in both the table and the abstract.
  3. [Section V-A, Table I] The core premise is that simulation-output consistency reliably identifies correct code, but the paper's own data show the signal is far from clean: 1135 of 5561 candidates (20.4%) that have a consistent partner are incorrect, while only 6 correct candidates lack consistency. The paper should report precision and recall of the consistency signal for each model and explicitly discuss how a 20% false-positive rate among "consistent" candidates interacts with cluster ranking. Table II's top-cluster accuracy (107/113 for GPT-4o) is reported for one model only and does not establish the reliability of the premise across the four LLMs claimed in the paper.
  4. [Section III-C, Section V-E] The CoT reference signals are generated by the same LLM that produces both the code candidates and the testbench, creating a self-referential selection loop: the arbiter may favor outputs that agree with its own prior predictions rather than with ground truth. The paper does not measure how often the CoT reference actually matches the ground-truth behavior on the disputed test cases, nor does it ablate the effect of using a fixed or human-written reference. Given that RQ5 is presented as a validation of the CoT mechanism, please provide this evidence or explicitly discuss the circularity risk as a limitation.
minor comments (6)
  1. [Section IV and Section V-C] The number of experimental repetitions is inconsistent: Section IV says the experiment is repeated 5 times, while Section V-C says it was repeated 10 times. Clarify which repetition count applies to which result.
  2. [Table III] The table uses the label "LLama" for Llama-3-70B; please use the official spelling. Also define "CoT Reference" in the caption, and state explicitly that CoT was not applied to Llama-3 and CodeV-Qwen because they could not produce formalized JSON output.
  3. [Table II] The parenthesized numbers in the second and third columns, e.g., "6(8)" and "0(1)", are not explained in the caption. Please describe what the values in parentheses represent (apparently the counts including lower-ranked clusters?).
  4. [Figure 5] The legend entries "Baseline Avg", "Our Method Avg", and "Case Loss Avg" are not explained in the caption; state that Case Loss refers to the case-wise consistency function defined in Section V-D.
  5. [Abstract] The abstract contains a typo: "passl1" should be "pass@1" (pass@1).
  6. [Section V-B, Equation (4)] The pass@k baseline formula appears to have a formatting issue: the expression should clearly denote the binomial-coefficient ratio used in the Codex pass@k estimator. Please ensure the mathematical notation is unambiguous and consistent with the definition in [10].

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VRank's reported gains are measured against external human-authored testbenches, not derived from its own consistency signal.

full rationale

The paper's central claim is an empirical pass@1 improvement on VerilogEval-Human, an external human-authored benchmark. The selection signal (simulation-output consistency on LLM-generated testbenches and CoT reference outputs) is not used as the ground truth for the reported result; the reported pass@1 is computed against the benchmark's ground-truth testbenches, as stated in Section IV: 'All simulations of LLM-generated tests and ground truth testbenches (only for validating our top pick) are carried out on iverilog.' The consistency metric R(c) = n - sum over c' of l_strict(c, c') is defined purely over candidate outputs and the LLM-generated testbench; it does not encode the target pass@1 metric. The paper validates the consistency signal against correctness in RQ1 (Table I) rather than assuming it, acknowledging the signal is noisy (1135 of 5561 candidates with consistency are incorrect). The CoT reference signal is also generated by the same LLM, but it is scored on the human benchmark, and Table III reports that it adds only 0.6-1.2 percentage points for the two models tested, so the central improvement does not reduce to the self-generated reference. No parameter is fitted to the target data and then reported as a prediction. There is a self-citation to [24] for the testbench generation technique, but that citation is not load-bearing: the framework's effectiveness is established by external benchmark results, not by the cited work's claims. No circular step could be identified; concerns about statistical significance (no confidence intervals for the 10.5% gain) and the noise in the consistency signal are correctness risks, not circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities; its dependence is on three empirical assumptions and several hand-chosen hyperparameters.

free parameters (4)
  • th% = 80
    CoT agreement threshold for swapping top clusters; no sensitivity analysis reported.
  • x = 5
    Number of CoT reasoning attempts; no sensitivity analysis reported.
  • m = >=10
    Number of test cases in the LLM-generated testbench; chosen heuristically.
  • n = up to 50
    Number of code candidates sampled; experiment sweeps 5 to 50.
assumptions (3)
  • domain assumption Correct solutions show more simulation-output consistency than incorrect solutions.
    Validated empirically in Table I, but not guaranteed; 20.4% of consistent candidates are incorrect.
  • domain assumption An LLM-generated testbench with at least 10 test cases and no reference outputs is sufficient to distinguish functionally different implementations.
    Invoked in Section III-A; testbench quality is not directly validated.
  • domain assumption Zero-shot CoT reasoning from the same LLM yields correct reference outputs for disputed test cases.
    Invoked in Section III-C; depends on the LLM's ability to reason about hardware behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VRank: Enhancing Verilog Code Generation from Large Language Models via Self-Consistency." pith.science (2026). https://pith.science/paper/ELEPNCSN

@misc{pith2026250200028,
  author       = {Pith},
  title        = {Pith review of: VRank: Enhancing Verilog Code Generation from Large Language Models via Self-Consistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ELEPNCSN}},
  note         = {Machine review of arXiv:2502.00028}
}
read the original abstract

Large Language Models (LLMs) have demonstrated promising capabilities in generating Verilog code from module specifications. To improve the quality of such generated Verilog codes, previous methods require either time-consuming manual inspection or generation of multiple Verilog codes, from which the one with the highest quality is selected with manually designed testbenches. To enhance the generation efficiency while maintaining the quality of the generated codes, we propose VRank, an automatic framework that generates Verilog codes with LLMs. In our framework, multiple code candidates are generated with LLMs by leveraging their probabilistic nature. Afterwards, we group Verilog code candidates into clusters based on identical outputs when tested against the same testbench, which is also generated by LLMs. Clusters are ranked based on the consistency they show on testbench. To determine the best candidate, Chain-of-Thought is further applied to select the best candidate from the top-ranked clusters. By systematically analyzing diverse outputs of generated codes, VRank reduces errors and enhances the overall quality of the generated Verilog code. Experimental results on the VerilogEval-Human benchmark demonstrate a significant 10.5% average increase in functional correctness (passl1) across multiple LLMs, demonstrating VRank's effectiveness in improving the accuracy of automated hardware description language generation for complex design tasks.

Figures

Figures reproduced from arXiv: 2502.00028 by the authors.

Figure 1
Figure 1. Comparison between (a) direct sampling, (b) debugging, and (c) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The outline of VRank. Our framework contains three major steps. (a) Execution-based clustering, (b) Cluster ranking, and (c) CoT decision. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. generating Verilog code candidates from LLM [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Functional correctness increase as # Samples increase [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Detail taxonomy pass rate of GPT-4o the CoT reasoning’s impact, we generated a taxonomy of the VerilogEval-Human benchmark inputs, as shown in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 23 canonical work pages

  1. [1]

    Competition-level code generation with AlphaCode,

    Y . Li et al. , “Competition-level code generation with AlphaCode,” Science, vol. 378, no. 6624, pp. 1092–1097, Dec. 9, 2022. arXiv: 2203. 07814[cs]

    work page 2022

  2. [2]

    K. Xu, G. L. Zhang, X. Yin, C. Zhuo, U. Schlichtmann, and B. Li, Automated c/c++ program repair for high-level synthesis via large language models, Jul. 4, 2024. arXiv: 2407.03889[eess]

    arXiv 2024

  3. [3]

    Machine learning in advanced IC design: A methodological survey,

    T. Chen, G. L. Zhang, B. Yu, B. Li, and U. Schlichtmann, “Machine learning in advanced IC design: A methodological survey,”IEEE Design & Test, vol. 40, no. 1, pp. 17–33, Feb. 2023, Conference Name: IEEE Design & Test

    work page 2023

  4. [4]

    K. Xu, R. Qiu, Z. Zhao, G. L. Zhang, U. Schlichtmann, and B. Li, LLM-aided efficient hardware design automation , Oct. 24, 2024. arXiv: 2410.18582[eess]

    arXiv 2024

  5. [5]

    Chang et al

    K. Chang et al. , ChipGPT: How far are we from natural language hardware design, Jun. 19, 2023. arXiv: 2305.14019[cs]

    arXiv 2023

  6. [6]

    R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, CorrectBench: Automatic testbench generation with functional self- correction using LLMs for HDL design , Nov. 13, 2024. arXiv: 2411. 08510[cs]

    work page 2024

  7. [7]

    Chip-chat: Chal- lenges and opportunities in conversational hardware design,

    J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-chat: Chal- lenges and opportunities in conversational hardware design,” in 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD) , Sep. 10, 2023, pp. 1–6. arXiv: 2305.13243[cs]

    arXiv 2023

  8. [8]

    BetterV: Controlled verilog generation with discriminative guidance,

    Z. Pei, H. Zhen, M. Yuan, Y . Huang, and B. Yu, “BetterV: Controlled verilog generation with discriminative guidance,” in Proceedings of the 41st International Conference on Machine Learning , ISSN: 2640-3498, PMLR, Jul. 8, 2024, pp. 40 145–40 153

    work page 2024

Show all 36 references
  1. [9]

    C.-T. Ho, H. Ren, and B. Khailany, VerilogCoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (AST)-based waveform tracing tool , version: 1, Aug. 15, 2024. arXiv: 2408.08927[cs]

    2024 arXiv

  2. [10]

    Chen et al

    M. Chen et al. , Evaluating large language models trained on code , Jul. 14, 2021. arXiv: 2107.03374[cs]

    2021 arXiv

  3. [11]

    Large language models for EDA: Future or mirage?

    Z. He and B. Yu, “Large language models for EDA: Future or mirage?” In Proceedings of the 2024 International Symposium on Physical Design, ser. ISPD ’24, New York, NY, USA: Association for Computing Machinery, Mar. 12, 2024, pp. 65–66

    2024

  4. [12]

    DA VE: Deriving automatically verilog from english,

    H. Pearce, B. Tan, and R. Karri, “DA VE: Deriving automatically verilog from english,” in Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD , ser. MLCAD ’20, New York, NY, USA: Association for Computing Machinery, Nov. 16, 2020, pp. 27–32

    2020

  5. [13]

    Liu et al

    M. Liu et al. , ChipNeMo: Domain-adapted LLMs for chip design , Apr. 4, 2024. arXiv: 2311.00176[cs]

    2024 arXiv

  6. [14]

    Benchmarking large language models for automated verilog RTL code generation,

    S. Thakur et al., “Benchmarking large language models for automated verilog RTL code generation,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), ISSN: 1558-1101, Apr. 2023, pp. 1–6

    2023

  7. [15]

    Zhao et al

    Y . Zhao et al. , CodeV: Empowering LLMs for verilog generation through multi-level summarization , version: 4, Jul. 20, 2024. arXiv: 2407.10424[cs]

    2024 arXiv

  8. [16]

    S. Liu, W. Fang, Y . Lu, Q. Zhang, H. Zhang, and Z. Xie, RTLCoder: Outperforming GPT-3.5 in design RTL generation with our open-source dataset and lightweight solution , Feb. 20, 2024. arXiv: 2312.08617[cs]

    2024 arXiv

  9. [17]

    Cui et al., OriGen:enhancing RTL code generation with code-to-code augmentation and self-reflection , Jul

    F. Cui et al., OriGen:enhancing RTL code generation with code-to-code augmentation and self-reflection , Jul. 23, 2024. arXiv: 2407.16237[cs]

    2024 arXiv

  10. [18]

    AutoVCoder: A systematic framework for automated verilog code generation using LLMs,

    M. Gao et al. “AutoVCoder: A systematic framework for automated verilog code generation using LLMs,” arXiv.org. (Jul. 21, 2024), (visited on 09/12/2024)

    2024

  11. [19]

    , GPT-4 technical report , Mar

    OpenAI et al. , GPT-4 technical report , Mar. 4, 2024. arXiv: 2303 . 08774[cs]

    2024

  12. [20]

    Improving large language model hardware generating quality through post-LLM search,

    K. Chang et al., “Improving large language model hardware generating quality through post-LLM search,”

  13. [21]

    Y .-D. Tsai, M. Liu, and H. Ren, RTLFixer: Automatically fixing RTL syntax errors with large language models , Feb. 7, 2024. arXiv: 2311. 16543[cs]

    2024

  14. [22]

    Retrieval-augmented generation for knowledge- intensive NLP tasks,

    P. Lewis et al. , “Retrieval-augmented generation for knowledge- intensive NLP tasks,” in Advances in Neural Information Processing Systems, vol. 33, Curran Associates, Inc., 2020, pp. 9459–9474

    2020

  15. [23]

    Thakur, J

    S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, AutoChip: Automating HDL generation using LLM feedback , Jun. 4,

  16. [24]

    Au- toBench: Automatic testbench generation and evaluation using LLMs for HDL design,

    R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, “Au- toBench: Automatic testbench generation and evaluation using LLMs for HDL design,” in Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD , ser. MLCAD ’24, New York, NY, USA: A...

    2024

  17. [25]

    Self-consistency improves chain of thought reason- ing in language models,

    X. Wang et al. , “Self-consistency improves chain of thought reason- ing in language models,” presented at the The Eleventh International Conference on Learning Representations, Sep. 29, 2022

    2022

  18. [26]

    Segmental minimum bayes-risk ASR voting strategies,

    V . Goel, S. Kumar, and W. Byrne, “Segmental minimum bayes-risk ASR voting strategies,” in Sixth International Conference on Spoken Language Processing, Citeseer, 2000

    2000

  19. [27]

    A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER),

    J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER),” in 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, IEEE, 1997, pp. 347–354

    1997

  20. [28]

    Minimum bayes-risk decoding for statistical machine translation,

    S. Kumar and B. Byrne, “Minimum bayes-risk decoding for statistical machine translation,” in Proceedings of the Human Language Technol- ogy Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 , 2004, pp. 169–176

    2004

  21. [29]

    Bleu: A method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , P. Isabelle, E. Charniak, and D. Lin, Eds., Philadelphia, Pennsylvania...

    2002

  22. [30]

    Wei et al

    J. Wei et al. , Chain-of-thought prompting elicits reasoning in large language models, Jan. 10, 2023. arXiv: 2201.11903[cs]

    2023 arXiv

  23. [31]

    VerilogEval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , 2023

    2023

  24. [32]

    Steveicarus/iverilog

    S. Williams. “Steveicarus/iverilog.” original-date: 2008-05- 12T16:57:52Z. (Sep. 17, 2024), [Online]. Available: https : //github.com/steveicarus/iverilog (visited on 09/18/2024)

    2008

  25. [33]

    Huggingface transformers

    Huggingface. “Huggingface transformers.” (), [Online]. Available: https: //huggingface.co/docs/transformers/index (visited on 09/22/2024)

    2024

  26. [34]

    Dubey et al

    A. Dubey et al. , The llama 3 herd of models , Aug. 15, 2024. arXiv: 2407.21783[cs]

    2024 arXiv

  27. [35]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,

    J. Lin et al., “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,” Proceedings of Machine Learning and Systems, vol. 6, pp. 87–100, May 29, 2024

    2024

  28. [2024]

    arXiv: 2311.04887[cs]

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.