Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Paradigm-Based Automatic HDL Code Generation Using LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that decomposing Verilog generation into expert-style paradigm blocks and a two-phase reuse workflow raises testbench pass rates by up to 14.7 percentage points on VerilogEval-human.

desk verdict A plausible LLM-based Verilog generation workflow, but the headline Pass@k gains are not directly measured—the method was run only on tasks the baseline failed, so the claimed improvements rest on an untested hybrid evaluation. read the letter →

arxiv 2501.12702 v1 pith:XOKZEZBD submitted 2025-01-22 cs.PL

Wenhao Sun , Bing Li , Grace Li Zhang , Xunzhao Yin , Cheng Zhuo , Ulf Schlichtmann This is my paper · ORCID
classification cs.PL
keywords Veriloggenerationhardwaredescriptionlanguagelargemodelshallucinationmitigationparadigmblockschain-of-thoughttwo-phaseworkflowPass@k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the main obstacle to LLM-generated Verilog is not lack of hardware knowledge but the models' weakness at multi-step tasks, and that a human-expert-inspired workflow can fix this without fine-tuning, extra databases, human oversight, or testbench feedback. The method decomposes generation into paradigm blocks that extract an information list, convert it into a structured form, and then write code, with a two-phase loop that reuses the most promising intermediate lists. On the VerilogEval-human benchmark, it reports gains of 4.7, 11.0, and 14.7 percentage points in Pass@1, Pass@5, and Pass@10 over single-conversation generation with GPT-4, and larger relative gains with GPT-4o-mini. A sympathetic reader would care because training-free improvements in hardware code generation bear directly on speeding up custom chip design.

What carries the argument

The machinery is the paradigm block, a small prompt-and-script pipeline that mirrors human design practice: information extraction into an explicit list, conversion into a machine-friendly intermediate format, and code generation from that format. SEQU (sequential) routes through a state-transition table and sequential always blocks, COMB (combinational) routes through a JSON truth table simplified by PyEDA into a sum-of-products expression, and BEHAV is a general fallback that reuses the best information lists. The two-phase workflow carries the argument by making intermediate artifacts reusable and ranked by testbench pass rate, so that a limited number of simulation rounds is spent on the most promising reasoning traces.

What would settle it

Run the proposed workflow on every task in VerilogEval (or a random sample that includes the easy tasks), count testbench passes directly, and compare Pass@k with the baseline on the same full task set; if the gains vanish or easy-task regressions offset hard-task gains, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that LLM hallucination in HDL generation is largely a multi-step reasoning failure, and that structuring the task into the same division-of-labor a human designer would use makes generated Verilog functionally correct more often. Concretely, the paper's paradigm blocks force the model first to spell out every input-output relation explicitly, then to cast that information into a state-transition table for sequential logic or a JSON truth table for combinational logic, then to produce code from those artifacts; an external tool, PyEDA, performs the logic simplification LLMs do poorly. A two-phase workflow then treats the testbench pass rate as a quality signal for the intermediate information lists, keeps the top candidates, and reuses them in a general BEHAV block when the specialized blocks fail. The paper reports that this pipeline outperforms direct generation and a testbench-feedback baseline, and that classification of circuit type improves when based on naively generated code rather than on the specification text alone.

Load-bearing premise

The headline gains assume that running the new workflow only on tasks the baseline already failed, and carrying over baseline scores for the rest, estimates full-dataset performance; the paper never checks whether the structured workflow could regress on tasks the baseline solved easily.

Editorial extensions

If this is right

  • On VerilogEval-human, the method beats direct one-shot generation at every Pass@k tested, for both GPT-4 and GPT-4o-mini.
  • On VerilogEval-machine, the method lifts GPT-4 Pass@5 and Pass@10 by more than five percentage points, though the combinational-logic procedure alone shows no gain there.
  • In the direct comparison against AutoChip on a hard-task subset, the method reaches 50 percent versus 40 percent on sequential logic and 70 percent versus 0 percent on combinational logic at Pass@10.
  • Circuit-type classification is more accurate when the LLM judges the type from its own first-draft Verilog code (98.3 percent) than from the specification text alone (94.9 percent) on the human-written set.
  • The hardest tasks, with testbench error rates above 0.8, barely shrink, so the method improves moderate errors more than extreme ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct full-dataset run, rather than the hard-task-only evaluation with baseline carryover, could plausibly change the headline numbers; the paper does not report one.
  • Because the pipeline is orthogonal to fine-tuning and retrieval, combining it with either could push pass rates further; the paper does not test this.
  • The testbench pass rate as a quality score for intermediate artifacts is a transferable idea: any generation task with a cheap verifier could rank and reuse its intermediate states the same way.
  • The gains attributed to the paradigms may partly come from simply spending more tokens and more simulation rounds on hard tasks, since the baseline gets one conversation per task.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training-free, human-expert-inspired framework for automatic Verilog generation with LLMs. The method classifies each specification as sequential, combinational, or general, then routes it to one of three paradigm blocks (SEQU, COMB, BEHAV). Each block decomposes generation into information extraction, intermediate formatted artifacts (truth tables, state-transition tables, component lists), and code synthesis, with PyEDA used for combinational simplification. A two-phase multi-round workflow with fail-safe and short-cut mechanisms is introduced to improve testbench pass rates under a fixed number of verification runs. Experiments on VerilogEval report Pass@k improvements over a single-conversation baseline for GPT-4 and GPT-4o-mini, plus a comparison with AutoChip.

Significance. If the reported gains are validated, the paper would make a useful contribution: it offers a practical, training-free alternative to fine-tuning and retrieval-augmented generation for HDL code generation, with a clear and well-structured workflow. The paper is also commendable for its detailed qualitative analysis of failure modes, including examples of information-list quality affecting downstream code correctness. However, the headline quantitative claims are currently not supported as stated because the evaluation on the main benchmark is conducted only on baseline-failed hard tasks, and the full-dataset Pass@k numbers appear to be a hybrid that assumes no regression on easy tasks. The lack of statistical uncertainty, the use of the test set for parameter selection, and the small AutoChip comparison further weaken the evidence. The central idea remains defensible, but the evaluation needs substantial revision before the stated conclusions can be accepted.

major comments (4)
  1. [Section IV, Table I] The main experimental claim is not a direct full-dataset measurement. The paper states: 'To reduce token costs, we first conducted the baseline experiments and then selected the hard tasks, for which the baseline method could not generate valid code to pass the testbench after 10 retries. The hard tasks are then processed by the proposed framework.' Yet Table I reports Pass@k for the FULL row against a baseline computed over the full dataset. If the FULL row is a hybrid (baseline results on easy tasks plus proposed results on hard tasks), it assumes without any evidence that the proposed workflow does not regress on tasks the baseline already solved. The workflow adds classification, multi-step prompting, and re-ranking, all of which can fail on easy tasks. If FULL instead reports results only on the hard-task subset, then comparing it with a full-dataset baseline is invalid. The paper must clarify the exact composition of the FULL row and provide results from running the proposed method on the full dataset, or at minimum on the easy-task subset, with the combined results reported transparently.
  2. [Section IV, Eq. (1)] The Pass@k estimator in Eq. (1) assumes that the n code samples for a task are exchangeable draws from a single sampling distribution. In the proposed two-phase workflow, the n=10 samples are not i.i.d.: phase I samples come from SEQU/COMB, phase II reuses selected information lists and switches to BEHAV, and fail-safe and short-cut modes allocate the remaining testbench executions differently depending on earlier outcomes. Applying the standard Pass@k formula to samples generated by this adaptive procedure is not methodologically justified, and the reported Pass@1/5/10 numbers may therefore be biased in an unknown direction. The authors should either use a protocol that draws n independent completions per task under exactly the workflow being evaluated, or explicitly derive and justify an estimator appropriate for this adaptive sampling scheme.
  3. [Section IV, Fig. 9 and Table I] The search parameters (N1, N2, N3), the short-cut threshold W, and the maximum iteration Smax appear to be tuned on the same VerilogEval test set. In particular, Fig. 9 compares configurations (5,3,2) and (7,2,1) on the same tasks and then selects (7,2,1) for the final reported results. This is a form of test-set tuning that can inflate the reported gains, especially combined with the small per-task sample size (n=10). The authors should either evaluate on a held-out split of VerilogEval (or a separate benchmark), or report the full selection procedure and the variance of results across multiple runs or random seeds. Without such information, the headline improvements of 4.7%, 11.0%, and 14.7% cannot be taken at face value.
  4. [Section IV, Table II] The comparison with AutoChip is based on only ten randomly selected tasks for each circuit type, with no random seed, no confidence intervals, and no breakdown of whether the selected tasks are representative. For a claim that the proposed method 'outperforms AutoChip, particularly in combinational logic,' this evidence is too thin. The comparison also needs to ensure that both methods are given the same computational budget (e.g., same total number of testbench executions and same number of LLM calls). Table II should report the full task list or a larger benchmark subset, and the authors should state the exact protocol used for AutoChip, including how its iteration count n=10 is aligned with the proposed method's Ntotal.
minor comments (5)
  1. [Section IV, Table I] The label 'FULL' is ambiguous: it could mean the full dataset, the full set of all generated samples, or the hard-task subset under the full workflow. Please define explicitly in the table caption or the text.
  2. [Section IV, Fig. 10] The x-axis label 'Percentage of Error Rate Intervals' is confusing; the x-axis appears to represent intervals of testbench error rates, while the y-axis represents configurations. Consider renaming the axis and adding a note that the proportions are computed over hard tasks only.
  3. [Section IV, paragraph 3] The statement that 'the COMB procedure did not show any improvement in the VerilogEval-machine dataset' is consistent with Table I, but it would help to state explicitly that this means the reported FULL improvement on VerilogEval-machine comes entirely from SEQU and BEHAV, which is a nontrivial observation about where the gains originate.
  4. [Section III, Fig. 2] The paradigm block names are rendered as 'SEQU', 'COMB', and 'BEHA V' in the figure and text. The inconsistent spacing in 'BEHA V' should be fixed to 'BEHAV' throughout for readability.
  5. [References] Reference [9] duplicates the content of reference [2]; please merge or renumber. Also, the access date format for reference [31] is inconsistent with the other entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the method and its evaluation are empirical; the hard-task-only evaluation raises validity concerns but no derivation reduces to its inputs.

full rationale

This is an empirical LLM-prompting paper, not a mathematical derivation. The method (paradigm blocks, type classifier, two-phase workflow) is constructed from external references to human design methodology and tools such as PyEDA, and the central evaluation is Pass@k measured by running Icarus Verilog testbenches. No equation equates an output to an input by definition. The only self-citations (e.g., [1], [6], [7], [8]) are background or tooling citations and are not load-bearing. The closest issue is that the proposed framework was applied only to hard tasks, and search parameters (N1,N2,N3,W) were selected after comparing two configurations on the same benchmark. These are evaluation-validity and overfitting concerns, not circularity: the reported pass rates are actual testbench outcomes, not quantities forced by a fitted input or by a self-citation chain. Under the stated rules, they therefore do not raise the circularity score.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on workflow hyperparameters tuned on the test set and on assumptions about LLM reliability in intermediate steps, but introduces no new physical or conceptual entities. The paradigm blocks are software templates, not invented entities with independent falsifiable handles.

free parameters (5)
  • N1 = 7
    Number of testbench executions allotted to phase I; part of the (7,2,1) configuration chosen after comparing with (5,3,2) on the same benchmark.
  • N2 = 2
    Number of top information-list candidates forwarded to phase II; tuned together with N1 and N3.
  • N3 = 1
    Number of BEHAV executions in phase II for the final loop; part of the (7,2,1) configuration.
  • W (short-cut threshold) = 0.95
    Pass-rate threshold that triggers short-cut mode; chosen by hand.
  • Ef (format retry limit) = 10
    Maximum retries when intermediate format errors occur; set by hand without sensitivity analysis.
assumptions (6)
  • domain assumption LLMs can classify circuits as combinational or sequential based on generated code.
    Table III reports 98-100% accuracy on VerilogEval; residual misclassification can route a task to the wrong paradigm block and degrade output.
  • domain assumption The generated information lists faithfully capture specification semantics, including timing requirements.
    Section III-A treats the information list as the input to all later steps; Fig. 11 shows that incomplete or wrong lists cause failed code.
  • domain assumption The three-always-block coding style is a valid target for sequential circuits.
    Section III-A forces sequential code into the three-always-block FSM style, which is standard but may not fit every sequential specification.
  • standard math PyEDA simplifies truth tables correctly.
    Section III-A relies on PyEDA to produce sum-of-products expressions; this is assumed correct without verification in the paper.
  • domain assumption VerilogEval testbenches are correct and sufficient.
    Pass rate is measured against these testbenches; an incorrect or incomplete testbench would invalidate the functional correctness measure.
  • standard math The Pass@k estimator with n=10 samples is a valid measure.
    Equation (1) is the standard estimator, but n=10 leads to high variance and the paper does not provide confidence intervals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Paradigm-Based Automatic HDL Code Generation Using LLMs." pith.science (2026). https://pith.science/paper/XOKZEZBD

@misc{pith2026250112702,
  author       = {Pith},
  title        = {Pith review of: Paradigm-Based Automatic HDL Code Generation Using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOKZEZBD}},
  note         = {Machine review of arXiv:2501.12702}
}
read the original abstract

While large language models (LLMs) have demonstrated the ability to generate hardware description language (HDL) code for digital circuits, they still face the hallucination problem, which can result in the generation of incorrect HDL code or misinterpretation of specifications. In this work, we introduce a human-expert-inspired method to mitigate the hallucination of LLMs and enhance their performance in HDL code generation. We begin by constructing specialized paradigm blocks that consist of several steps designed to divide and conquer generation tasks, mirroring the design methodology of human experts. These steps include information extraction, human-like design flows, and the integration of external tools. LLMs are then instructed to classify the type of circuit in order to match it with the appropriate paradigm block and execute the block to generate the HDL codes. Additionally, we propose a two-phase workflow for multi-round generation, aimed at effectively improving the testbench pass rate of the generated HDL codes within a limited number of generation and verification rounds. Experimental results demonstrate that our method significantly enhances the functional correctness of the generated Verilog code

Figures

Figures reproduced from arXiv: 2501.12702 by the authors.

Figure 1
Figure 1. Comparison of Training-Free Methods: na¨ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The proposed paradigm blocks SEQU, COMB, and BEHAV are [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Example of information list generated by LLMs for a sequential logic [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Example of the state-transition table generated by LLM for a sequential [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Example of the truth table in JSON generated by LLM for a [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Example of a always block generated by LLM for task ece241 2013 q12. address this challenge, we propose a two-phase workflow, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 9
Figure 9. Figure 9: Comparison of the Pass@k under GPT-4 between configurations, [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Comparison of testbench error rate distributions of GPT-4 for hard [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Comparison of snippets of information lists between two attempts [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Progressive Approach to Synthesizable RTL Design Generation Using LLMs

    cs.AR 2026-07 conditional novelty 6.0 of 10

    VeriRefine boosts LLM-generated RTL correctness to 94.0% on RTLLM v2.0 and 98.1% on VerilogEval-Human v2 by refining and auditing a per-signal intermediate representation before code generation.

  2. VeriOpt: PPA-Aware High-Quality Verilog Generation via Multi-Role LLMs

    cs.SE 2025-07 conditional novelty 5.0 of 10

    A multi-role LLM prompting framework with PPA-aware in-context learning reports 25/29 functional correctness on RTLLM and up to 88% power, 76% area, and 73% timing gains over its own baseline.

  3. ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Model

    cs.AR 2025-04 conditional novelty 5.0 of 10

    A 7B Verilog generation model with a verified 5K dataset, two-stage training, and an adaptive reasoning router reaches 57.8% pass@1 on VerilogEval-human, 10.4 points over the prior best open-source Verilog-specific model.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages · cited by 3 Pith papers

  1. [1]

    Llm- aided efficient hardware design automation,

    K. Xu, R. Qiu, Z. Zhao, G. L. Zhang, U. Schlichtmann, and B. Li, “Llm- aided efficient hardware design automation,” 2024

    work page 2024

  2. [2]

    Machine learning in advanced ic design: A methodological survey,

    T. Chen, G. L. Zhang, B. Yu, B. Li, and U. Schlichtmann, “Machine learning in advanced ic design: A methodological survey,” IEEE Design & Test, vol. 40, no. 1, pp. 17–33, 2023

    work page 2023

  3. [3]

    Prompting large language model for machine translation: A case study,

    B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” inInternational Conference on Machine Learning (ICML), 2023, pp. 41 092–41 110

    work page 2023

  4. [4]

    Tidybot: Personalized robot assis- tance with large language models,

    J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personalized robot assis- tance with large language models,” Autonomous Robots, vol. 47, no. 8, pp. 1087–1102, 2023

    2023

  5. [5]

    The programmer’s assistant: Conversational interaction with a large language model for software development,

    S. I. Ross, F. Martinez, S. Houde, M. Muller, and J. D. Weisz, “The programmer’s assistant: Conversational interaction with a large language model for software development,” in International Conference on Intelli- gent User Interfaces (IUI), 2023, pp. 491–514

    work page 2023

  6. [6]

    Autobench: Automatic testbench generation and evaluation using llms for hdl design,

    R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, “Autobench: Automatic testbench generation and evaluation using llms for hdl design,” in ACM/IEEE International Symposium on Machine Learning for CAD , 2024

    work page 2024

  7. [7]

    Au- tomated c/c++ program repair for high-level synthesis via large language models,

    K. Xu, G. L. Zhang, X. Yin, C. Zhuo, U. Schlichtmann, and B. Li, “Au- tomated c/c++ program repair for high-level synthesis via large language models,” in Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, 2024

    work page 2024

  8. [8]

    Cor- rectbench: Automatic testbench generation with functional self-correction using llms for hdl design,

    R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, “Cor- rectbench: Automatic testbench generation with functional self-correction using llms for hdl design,” in Design, Automation and Test in Europe Conference and Exhibition (DATE), 2025

    work page 2025

Show all 34 references
  1. [9]

    Machine learning in advanced ic design: A methodological survey,

    T. Chen, G. L. Zhang, B. Yu, B. Li, and U. Schlichtmann, “Machine learning in advanced ic design: A methodological survey,” IEEE Design & Test, vol. 40, pp. 17–33, 2023

    2023

  2. [10]

    Benchmarking large language models for automated Verilog RTL code generation,

    S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan- Gavitt, and S. Garg, “Benchmarking large language models for automated Verilog RTL code generation,” in Design, Automation and Test in Europe Conference and Exhibition (DATE), 2023, pp. 1–6

    2023

  3. [11]

    Code- halu: Code hallucinations in llms driven by execution-based verification,

    Y . Tian, W. Yan, Q. Yang, Q. Chen, W. Wang, Z. Luo, and L. Ma, “Code- halu: Code hallucinations in llms driven by execution-based verification,” arXiv preprint arXiv:2405.00253, 2024

    2024 arXiv

  4. [12]

    Chip-chat: Challenges and opportunities in conversational hardware design,

    J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-chat: Challenges and opportunities in conversational hardware design,” in ACM/IEEE Workshop on Machine Learning for CAD (MLCAD), 2023, pp. 1–6

    2023

  5. [13]

    ChipNeMo: Domain- adapted llms for chip design,

    M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu et al. , “ChipNeMo: Domain- adapted llms for chip design,” arXiv preprint arXiv:2311.00176, 2023

    2023 arXiv

  6. [14]

    RTLcoder: Outperforming gpt-3.5 in design RTL generation with our open-source dataset and lightweight solution,

    S. Liu, W. Fang, Y . Lu, Q. Zhang, H. Zhang, and Z. Xie, “RTLcoder: Outperforming gpt-3.5 in design RTL generation with our open-source dataset and lightweight solution,” arXiv preprint arXiv:2312.08617, 2023

    2023 arXiv

  7. [15]

    VeriGen: A large language model for Verilog code generation,

    S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri, and S. Garg, “VeriGen: A large language model for Verilog code generation,” ACM Transactions on Design Automation of Electronic Systems , vol. 29, no. 3, pp. 1–31, 2024

    2024

  8. [16]

    BetterV: Con- trolled verilog generation with discriminative guidance,

    Z. Pei, H.-L. Zhen, M. Yuan, Y . Huang, and B. Yu, “BetterV: Con- trolled verilog generation with discriminative guidance,” arXiv preprint arXiv:2402.03375, 2024

    2024 arXiv

  9. [17]

    Data is all you need: Finetuning LLMs for chip design via an automated design-data augmentation framework,

    K. Chang, K. Wang, N. Yang, Y . Wang, D. Jin, W. Zhu, Z. Chen, C. Li, H. Yan, Y . Zhou et al. , “Data is all you need: Finetuning LLMs for chip design via an automated design-data augmentation framework,” arXiv preprint arXiv:2403.11202, 2024

    2024 arXiv

  10. [18]

    GPT-4 tech- nical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “GPT-4 tech- nical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  11. [19]

    Rethinking the role of demonstrations: What makes in- context learning work?

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in- context learning work?” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 11 048–11 064

    2022

  12. [20]

    Generation- augmented retrieval for open-domain question answering,

    Y . Mao, P. He, X. Liu, Y . Shen, J. Gao, J. Han, and W. Chen, “Generation- augmented retrieval for open-domain question answering,” inInternational Joint Conference on Natural Language Processing (IJCNLP) , 2021, pp. 4089–4100

    2021

  13. [21]

    GPT4AIGChip: Towards next-generation ai accelerator design automation via large language models,

    Y . Fu, Y . Zhang, Z. Yu, S. Li, Z. Ye, C. Li, C. Wan, and Y . C. Lin, “GPT4AIGChip: Towards next-generation ai accelerator design automation via large language models,” in IEEE/ACM International Conference on Computer Aided Design (ICCAD), 2023, pp. 1–9

    2023

  14. [22]

    RTLfixer: Automatically fixing RTL syntax errors with large language models,

    Y . Tsai, M. Liu, and H. Ren, “RTLfixer: Automatically fixing RTL syntax errors with large language models,” arXiv preprint arXiv:2311.16543 , 2023

    2023 arXiv

  15. [23]

    HDLdebugger: Streamlining HDL debugging with large language models,

    X. Yao, H. Li, T. H. Chan, W. Xiao, M. Yuan, Y . Huang, L. Chen, and B. Yu, “HDLdebugger: Streamlining HDL debugging with large language models,” arXiv preprint arXiv:2403.11671, 2024

    2024 arXiv

  16. [24]

    Au- tochip: Automating HDL generation using LLM feedback,

    S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “Au- tochip: Automating HDL generation using LLM feedback,” arXiv preprint arXiv:2311.04887, 2023

    2023 arXiv

  17. [25]

    VerilogEval: Evaluating large language models for Verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: Evaluating large language models for Verilog code generation,” inIEEE/ACM Interna- tional Conference on Computer-Aided Design (ICCAD), 2023

    2023

  18. [26]

    Do large language models latently perform multi-hop reasoning?

    S. Yang, E. Gribovskaya, N. Kassner, M. Geva, and S. Riedel, “Do large language models latently perform multi-hop reasoning?” arXiv preprint arXiv:2402.16837, 2024

    2024 arXiv

  19. [27]

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process,

    T. Ye, Z. Xu, Y . Li, and Z. Allen-Zhu, “Physics of language models: Part 2.1, grade-school math and the hidden reasoning process,” arXiv preprint arXiv:2407.20311, 2024

    2024 arXiv

  20. [28]

    Chain-of-Thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-Thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 24 824–24 837

    2022

  21. [29]

    To believe or not to believe your llm,

    Y . A. Yadkori, I. Kuzborskij, A. Gy¨orgy, and C. Szepesv´ari, “To believe or not to believe your llm,” arXiv preprint arXiv:2406.02543, 2024

    2024 arXiv

  22. [30]

    Coding techniques in verilog for finite state machine designs in FPGA,

    V . Salauyou and Ł. Zabrocki, “Coding techniques in verilog for finite state machine designs in FPGA,” in Computer Information Systems and Industrial Management (CISIM). Springer, 2019, pp. 493–505

    2019

  23. [31]

    GPT-3.5 turbo: Language models,

    “GPT-3.5 turbo: Language models,” Accessed: 2022 [Online]. Available: https://platform.openai.com/docs/models/gpt-3-5-turbo

    2022

  24. [32]

    GPT-4o mini: advancing cost-efficient intelligence,

    “GPT-4o mini: advancing cost-efficient intelligence,” Ac- cessed: 2024 [Online]. Available: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  25. [33]

    Icarus Verilog: open-source Verilog more than a year later,

    S. Williams and M. Baxter, “Icarus Verilog: open-source Verilog more than a year later,” Linux Journal, vol. 2002, no. 99, p. 3, 2002

    2002

  26. [34]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.