Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

SVAgent: AI Agent for Hardware Security Verification Assertion

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SVAgent decomposes each security requirement into a chain of sub-questions, generating SystemVerilog assertions with functional and syntactic accuracy mostly above 90 percent across five LLMs, without fine-tuning.

desk verdict Useful prompt-decomposition framework for security SVA generation, but the undefined scoring metric makes the headline accuracy claims unverifiable as written. read the letter →

arxiv 2507.16203 v1 pith:6KLDLMYX submitted 2025-07-22 cs.CR cs.AIcs.ARcs.LG

Rui Guo , Avinash Ayalasomayajula , Henian Li , Jingbo Zhou , Sujan Kumar Saha , Farimah Farahmandi This is my paper
classification cs.CRcs.AIcs.ARcs.LG
keywords SystemVerilogassertionshardwaresecuritylargelanguagemodelsSVAgenerationassertion-basedverificationpromptengineeringrequirementdecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SVAgent is a prompting framework that uses a large language model (LLM) to automatically write SystemVerilog assertions (SVA)—formal checks that a hardware design obeys a required behavior—for security verification, without fine-tuning. Its central move is to decompose each high-level security requirement into a chain of small sub-questions, answer them with tightly scaffolded prompts, and then assemble the answers into a complete assertion file. On roughly five hundred designs and five different LLMs, the paper reports functional and syntactic accuracy usually above 90 percent and often 100 percent, while direct generation from the original requirement produces no fully usable SVA. It also claims that an engineer writes only 500–700 lines of prompt template per threat model, and that the generated assertions can replace manual assertion writing inside the SoFI vulnerability-assessment flow. If these results hold, a scarce, labor-intensive expert task in hardware security verification could become largely automated.

What carries the argument

The load-bearing mechanism is the decomposer–prompt-generator–reorganizer loop. The Decomposer takes a security requirement (anchored to a threat model from Trust-HUB or CWE) and splits it into a logically ordered chain of sub-questions that progress from basic design facts to per-asset assertion snippets. The Prompt Generator builds one prompt per sub-question from the previous sub-answer, three valid input-output examples, and one invalid example, so the LLM only ever reasons over a focused slice of the design. The Reorganizer then assembles the snippets into a complete SystemVerilog file with a fixed script, without relying on the LLM. For the state-transition threat model, the third-party tool Pyverilog is embedded to supply FSM analysis output to the chain.

What would settle it

Run the SVAgent-generated assertions through a model checker (for instance, JasperGold) on the same benchmarks with known inserted vulnerabilities; if the assertions pass on vulnerable designs or fail to flag them while hand-written reference assertions do, then the reported accuracy scores do not reflect real security-detection ability.

Watch

Extended reading notes

Core claim

The paper's central claim is that the main obstacle to LLM-generated SVA is not the model's raw capability but the way the task is presented: giving an LLM a full security requirement and a whole RTL design at once overloads its attention and invites hallucination and random answers. SVAgent addresses this by decomposing each requirement into a problem-solving chain of sub-questions—extract design basics, list assets, identify sensitive signals, then generate one assertion snippet per asset—each prompted with three valid and one invalid example, and finally reassembling the snippets with a deterministic local script. The authors report that while direct generation produces no fully usable SVA, the decomposition lifts functional and syntactic accuracy to values usually above 90 percent across GPT-4, Gemini-Pro, Claude 3, Meta AI, and Copilot, and suppresses run-to-run variation: more than 80 percent of generated code is identical or acceptable across five repeated runs. The finer the chain, the higher the accuracy, and model-specific syntax errors (such as Gemini-Pro's trailing backticks on constants) can be eliminated by adding a targeted sub-question. Integrated into SoFI, the framework generated assertions for two AES fault-injection security properties with 91.67% and 100% stated correctness.

Load-bearing premise

The load-bearing premise is that the paper's 0–100 scores measure semantic correctness of each generated assertion against the target vulnerability, not just similarity to the hand-crafted examples in the prompts.

Editorial extensions

If this is right

  • Verification engineers would only need to write a prompt template once per threat model (500–700 lines) and could then apply it to any number of designs, removing the per-design prompt engineering that frameworks like AutoSVA2 and ChIRAAG require.
  • Because SVAgent is model-agnostic and needs no fine-tuning or GPU clusters, teams without large compute budgets could generate security assertions with whatever LLM they already use.
  • A recurring model-specific syntax error can be fixed by inserting one targeted sub-question into the chain, which means assertion quality can be tuned by chain design rather than by switching models.
  • Within the SoFI vulnerability-assessment flow, the two tested AES fault-injection properties can have their assertions generated automatically with 91.67% and 100% reported correctness, replacing a manual expert step.
  • The high run-to-run consistency (over 80% identical or acceptable code across five runs) suggests the generated assertions would not silently change meaning between verification runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper only hints at is to apply the same decompose-generate-reorganize loop to other formal artifacts, such as functional coverage properties or security contracts, rather than only SVA assertions.
  • Because the paper's accuracy metric is its own scoring rubric rather than formal equivalence checking, a natural validation step is to model-check the generated assertions against the actual design and measure whether they flag planted faults.
  • Because the decomposition templates are tied to threat models, a further automation step would be to derive sub-question templates automatically from CWE entries or Trust-HUB descriptions, reducing the remaining 500–700 lines of engineer-written prompt per threat model.
  • The reported weakness with circuits that have too many I/O ports suggests that adding a dedicated sub-question to filter or group ports could be a cheap, testable improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SVAgent, an LLM-based framework that automatically generates SystemVerilog assertions (SVA) for hardware security verification. The framework decomposes a natural-language security requirement into a chain of fine-grained sub-questions, supplies prompts containing three valid and one invalid example, and then reassembles the resulting code snippets into a complete SVA file via an offline reorganization script. The authors evaluate SVAgent with five LLMs on Trust-HUB and Pyverilog benchmarks across four threat models, reporting functionality and syntax scores that are often above 80 percent, a consistency study against AutoSVA2, and an integration case study with the SoFI fault-injection assessment framework. The paper claims that SVAgent suppresses LLM hallucination and random answers, reduces engineer workload, and can replace manual security assertion writing.

Significance. If the evaluation were methodologically sound, the contribution would be practically valuable: security-oriented SVA is typically written by hand, and a prompt-engineering approach that avoids fine-tuning and per-design prompt redesign is attractive. The decomposition idea is reasonable, the use of a problem-solving chain is clearly described, and the consistency experiment over 16 circuits with five repetitions is a useful direction. The authors also explicitly acknowledge a limitation in the diversity of generated assertion patterns. However, the central quantitative claims rest on an undefined scoring protocol, a small and informally scored SoFI integration, and no released prompts, code, or per-assertion results. The significance is therefore conditional on the evaluation being made objective and reproducible.

major comments (4)
  1. [Section IV-B, Table II] The scoring mechanism for the 'Functionality' and 'Syntax' columns is never specified. The paper only states that 'We use a scoring mechanism (out of 100 points)' and that JasperGold was used as the model checker. It does not state whether each generated assertion was model-checked against the vulnerable design, whether the assertion was required to fail on a patched or secure version, how partial credit was assigned, or whether scores were produced by comparison with hand-written reference assertions or by an LLM judge. Without this protocol, the high numbers cannot be interpreted as detection accuracy: an assertion that is syntactically valid but never fires on the actual attack path could still receive a high score under a similarity-based rubric. Please define the rubric precisely, provide per-assertion outcomes on vulnerable and secure designs, and report how many designs and runs underlie each entry in Table II.
  2. [Section IV-B, Figure 7] The comparison labeled 'original requirement' versus 'problem-solving chain' is not controlled for the post-hoc correction stage described later in the same section. The authors state that after generating an SVA snippet they add a sub-question such as 'Check whether there are quotation marks at the end of all constants...' and delete them. Thus the reported syntax accuracy includes an additional correction stage that is not part of the initially described pipeline, and the reader cannot tell which improvement comes from decomposition versus from syntax-fixing prompts. Please run and report an ablation without the correction stage, or report the exact pipeline as evaluated, including all sub-questions used.
  3. [Section IV-C, SoFI integration] The SoFI experiment uses only two security properties, SP3.1 and SP3.2, and the sentence reporting 'correctness of the assertions generated for the intermediate state reached 91.67%' does not define the ground truth for correctness or state the number of assertions involved. Two properties are too few to support the claim that SVAgent 'has the potential to replace the process of manually writing SVAs' in existing frameworks. Please expand the SoFI evaluation or clearly label it as a preliminary case study and correspondingly soften the scalability conclusion.
  4. [Section IV-B, Table II] No variance or repetition information is given for the Table II scores, so the claimed 'universality' across LLMs is not statistically supported. The entries vary substantially by model and threat model, for example Copilot at 62.5 percent functionality on unused states and Gemini-Pro at 68.7 percent syntax on info leakage. Please report the number of designs per cell, the number of repeated generations, and either confidence intervals or the full distribution of per-design scores.
minor comments (5)
  1. [Algorithm 1] There is a typo in the algorithm pseudocode: 'Regonizer' should be 'Reorganizer', and the line breaks around 'SV A = Regonizer(lCodeSnippet)' should be cleaned up.
  2. [Section IV-A] The workload experiment mentions 450 circuit designs while the experimental setup says 'nearly 500 designs'; please clarify the exact number and define what counts as engineer-written lines of code.
  3. [Section IV-B] The Verilog constant example uses a typographic apostrophe in `3′b001`; use ASCII `3'b001`, and use consistent terminology for 'backticks' versus 'quotation marks' when describing the syntax error fixed by the added sub-question.
  4. [Section IV-B] The statement that 'all LLMs used the same prompts' should be accompanied by model version identifiers, temperature settings, and any sampling parameters, since those affect the reported consistency and accuracy numbers.
  5. [References] The web references for Trust-HUB, CWE, and the model pages lack access dates and version identifiers; please format them consistently with the rest of the bibliography.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: SVAgent's accuracy claims rest on external benchmarks rather than on fitted parameters or self-referential scoring; the only self-citations are background or integration references, and the underspecified scoring metric is a rigor concern, not circularity.

full rationale

The paper's derivation chain is: decompose natural-language security requirements into sub-questions via the Decomposer, prompt an LLM with few-show examples, collect SVA snippets, and reassemble them into a SystemVerilog file. The reported accuracy numbers are not produced by fitting parameters and then predicting those same parameters back; there are no fitted parameters at all. The few-shot examples in Figs. 4 and 5 encode the expected output style, but the benchmarks are external designs from TrustHUB, Pyverilog, and [43], and the threat models are standard CWEs, so the accuracy comparison is not self-referential. The author self-citations that do appear, such as [1] and [3] and possibly SoFI [16], are used for background context or as the integration target, not as the evidence that SVAgent's assertions are correct. The most serious weakness is that the paper never defines the 'scoring mechanism (out of 100 points)' used for the Functionality and Syntax columns, writing only: 'We use a scoring mechanism (out of 100 points) to evaluate the generated code, taking into account both functional/logical correctness and syntactic correctness.' Without a rubric, the reader cannot independently verify that the scores measure semantic vulnerability-detection power. However, an unspecified evaluation metric is a rigor or correctness gap, not a circularity: the paper does not define the score in terms of similarity to the prompt examples, nor does it fit a parameter and then 'predict' that same parameter. Under the hard rules requiring a specific exhibited reduction, no circular step can be identified. The score of 2 reflects the presence of minor non-load-bearing self-citations; the central derivation is self-contained against external benchmarks.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or mathematical entities. The main unstated inputs are the hand-crafted prompt templates and the evaluator's scoring rule, neither of which is independently verified.

free parameters (1)
  • k (number of valid examples per prompt) = 3
    Chosen as default in Algorithm 1; the paper does not report an ablation over different k values.
assumptions (3)
  • domain assumption LLMs can correctly perform each fine-grained sub-task (e.g., extracting signal lists, identifying unused states) when provided with a few hand-crafted examples.
    The whole SVAgent pipeline depends on this capability; Section III-C asserts that this improves accuracy but provides no controlled test beyond the final SVA accuracy.
  • domain assumption The Trust-HUB, Pyverilog, and [43] benchmarks are representative of real hardware security verification workloads.
    Section IV-B uses these benchmarks to claim generality; no evidence is given that they reflect the complexity of industrial designs.
  • domain assumption The scoring mechanism (out of 100) used to evaluate functionality and syntax is an accurate and unbiased measure of whether an SVA correctly detects the target vulnerability.
    Section IV-B mentions 'a scoring mechanism' without defining it; this is load-bearing for the accuracy claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SVAgent: AI Agent for Hardware Security Verification Assertion." pith.science (2026). https://pith.science/paper/6KLDLMYX

@misc{pith2026250716203,
  author       = {Pith},
  title        = {Pith review of: SVAgent: AI Agent for Hardware Security Verification Assertion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KLDLMYX}},
  note         = {Machine review of arXiv:2507.16203}
}
read the original abstract

Verification using SystemVerilog assertions (SVA) is one of the most popular methods for detecting circuit design vulnerabilities. However, with the globalization of integrated circuit design and the continuous upgrading of security requirements, the SVA development model has exposed major limitations. It is not only inefficient in development, but also unable to effectively deal with the increasing number of security vulnerabilities in modern complex integrated circuits. In response to these challenges, this paper proposes an innovative SVA automatic generation framework SVAgent. SVAgent introduces a requirement decomposition mechanism to transform the original complex requirements into a structured, gradually solvable fine-grained problem-solving chain. Experiments have shown that SVAgent can effectively suppress the influence of hallucinations and random answers, and the key evaluation indicators such as the accuracy and consistency of the SVA are significantly better than existing frameworks. More importantly, we successfully integrated SVAgent into the most mainstream integrated circuit vulnerability assessment framework and verified its practicality and reliability in a real engineering design environment.

Figures

Figures reproduced from arXiv: 2507.16203 by the authors.

Figure 1
Figure 1. There are many differences in the SVA generated by AutoSVA2 for the same design. This is the result of the LLM’s attention mechanism, hallucination, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. SVAgent Powered LLM-based SVA Auto-Generation Flow. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Decomposer examples concerning (a) unused states in FSM, and (b) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The prompt generation for sub − Questionn consists of three valid examples for sub − Questionn, one invalid example for sub − Questionn, sub − Questionn itself, and valid information from the answer of sub − Questionn−1 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: shows an example of the prompt used in the code generation stage. After obtaining all necessary design information and intermediate analysis results, the final step is to generate specific SVA code snippets in a structured and direct manner. The design of this prompt i…
Figure 6
Figure 6. Figure 6: Comparison of the amount of work required by engineers to write [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Average correctness of generated SVA from original requirement to [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Comparison of consistency of generated SVA. The smaller the [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Language to Logic: Bridging LLMs & Formal Representations for RTL Assertion Generation

    cs.CR 2026-04 unverdicted novelty 7.0 of 10

    ProofLoop achieves 93.7% syntax correctness and 82.0% functional correctness for SVA generation from natural language by combining retrieval, EDA tools, and up to three rounds of JasperGold formal feedback.

  2. CHARGE: Leveraging CWE Hierarchies for Hardware Security SystemVerilog Assertion Generation

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Using CWE hierarchy-aware LLM prompts, CHARGE generates security SVAs from unverified RTL, detecting 27 of 42 Hack@DAC bugs and one new key-reuse flaw.

Reference graph

Works this paper leans on

50 extracted references · 37 canonical work pages · cited by 2 Pith papers

  1. [1]

    Evolute: evaluation of look-up-table-based fine-grained ip redaction,

    Guo, Rui et al., “Evolute: evaluation of look-up-table-based fine-grained ip redaction,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE) . IEEE, 2023, pp. 1–6

    work page 2023

  2. [2]

    Soc security verification using property check- ing,

    Farzana, Nusrat et al., “Soc security verification using property check- ing,” in 2019 IEEE International Test Conference (ITC), 2019, pp. 1–10

    work page 2019

  3. [3]

    Formal approaches to hard- ware trust verification,

    F. Farahmandi, Y . Huang, and P. Mishra, “Formal approaches to hard- ware trust verification,” The Hardware Trojan War: Attacks, Myths, and Defenses, pp. 183–202, 2018

    work page 2018

  4. [4]

    Dynamically discovering likely program invariants to support program evolution,

    Ernst, Michael D et al. , “Dynamically discovering likely program invariants to support program evolution,” in Proceedings of the 21st international conference on Software engineering , 1999, pp. 213–224

    work page 1999

  5. [5]

    In-ide code generation from natural language: Promise and challenges,

    Xu, Frank F et al. , “In-ide code generation from natural language: Promise and challenges,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 31, no. 2, pp. 1–47, 2022

    work page 2022

  6. [6]

    Mathprompter: Mathematical reasoning using large language models,

    Imani, Shima et al., “Mathprompter: Mathematical reasoning using large language models,” arXiv preprint arXiv:2303.05398 , 2023

    arXiv 2023

  7. [7]

    Is chatgpt a general-purpose natural language processing task solver?

    Qin, Chengwei et al. , “Is chatgpt a general-purpose natural language processing task solver?” arXiv preprint arXiv:2302.06476 , 2023

    arXiv 2023

  8. [8]

    Emergent analogical reasoning in large language models,

    Webb, Taylor et al. , “Emergent analogical reasoning in large language models,” Nature Human Behaviour, vol. 7, no. 9, pp. 1526–1541, 2023

    work page 2023

Show all 50 references
  1. [9]

    Palm-e: An embodied multimodal language model,

    Driess, Danny et al. , “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378 , 2023

    2023 arXiv

  2. [10]

    Dynamic llm-agent network: An llm-agent col- laboration framework with agent team optimization,

    Liu, Zijun et al. , “Dynamic llm-agent network: An llm-agent col- laboration framework with agent team optimization,” arXiv preprint arXiv:2310.02170, 2023

    2023 arXiv

  3. [11]

    Executable code actions elicit better llm agents,

    Wang, Xingyao et al., “Executable code actions elicit better llm agents,” in Forty-first International Conference on Machine Learning , 2024

    2024

  4. [12]

    Llm-assisted generation of hardware assertions,

    Kande, Rahul et al. , “Llm-assisted generation of hardware assertions,” arXiv preprint arXiv:2306.14027 , 2023

    2023 arXiv

  5. [13]

    Divas: An llm-based end-to-end framework for soc security analysis and policy-based protection,

    Sudipta Paria et al., “Divas: An llm-based end-to-end framework for soc security analysis and policy-based protection,” 2023

    2023

  6. [14]

    Bugs in large language models generated code: An empirical study,

    Tambon, Florian et al., “Bugs in large language models generated code: An empirical study,” Empirical Software Engineering, vol. 30, no. 3, pp. 1–48, 2025

    2025

  7. [15]

    Explore what llm does not know in complex ques- tion answering,

    Lin, Xin et al. , “Explore what llm does not know in complex ques- tion answering,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 23, 2025, pp. 24 585–24 594

    2025

  8. [16]

    Sofi: Security property-driven vulnerability as- sessments of ics against fault-injection attacks,

    Wang, Huanyu et al. , “Sofi: Security property-driven vulnerability as- sessments of ics against fault-injection attacks,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 41, no. 3, pp. 452–465, 2021

    2021

  9. [17]

    A survey of microarchitectural timing attacks and countermeasures on contemporary hardware,

    Ge, Qian et al. , “A survey of microarchitectural timing attacks and countermeasures on contemporary hardware,” Journal of Cryptographic Engineering, vol. 8, pp. 1–27, 2018

    2018

  10. [18]

    A survey of microarchitectural side-channel vulnerabilities, attacks, and defenses in cryptography,

    Lou, Xiaoxuan et al. , “A survey of microarchitectural side-channel vulnerabilities, attacks, and defenses in cryptography,” ACM Computing Surveys (CSUR), vol. 54, no. 6, pp. 1–37, 2021

    2021

  11. [19]

    Em side channels in hardware security: Attacks and defenses

    He, Jiaji et al. , “Em side channels in hardware security: Attacks and defenses.” IEEE Des. Test, vol. 39, no. 2, pp. 100–111, 2022

    2022

  12. [20]

    Em side-channel attack on aes,

    Tehranipoor, Mark et al., “Em side-channel attack on aes,” in Hardware Security Training, Hands-on! Springer, 2023, pp. 163–181

    2023

  13. [21]

    Leveraging side-channel information for disassem- bly and security,

    Park, Jungmin et al., “Leveraging side-channel information for disassem- bly and security,” ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 16, no. 1, pp. 1–21, 2019

    2019

  14. [22]

    , Hardware security: a hands-on learning ap- proach

    Bhunia, Swarup et al. , Hardware security: a hands-on learning ap- proach. Morgan Kaufmann, 2018

    2018

  15. [23]

    Breaking and entering through the silicon,

    Helfmeier, Clemens et al., “Breaking and entering through the silicon,” in Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, 2013, pp. 733–744

    2013

  16. [24]

    Laser fault attack on physically unclonable functions,

    Tajik, Shahin et al. , “Laser fault attack on physically unclonable functions,” in 2015 workshop on fault diagnosis and tolerance in cryptography (FDTC). IEEE, 2015, pp. 85–96

    2015

  17. [25]

    Power supply glitch induced faults on fpga: An in-depth analysis of the injection mechanism,

    Zussa, Loic et al. , “Power supply glitch induced faults on fpga: An in-depth analysis of the injection mechanism,” in 2013 IEEE 19th International On-Line Testing Symposium (IOLTS) . IEEE, 2013, pp. 110–115

    2013

  18. [26]

    When clocks fail: On critical paths and clock faults,

    Agoyan, Michel et al. , “When clocks fail: On critical paths and clock faults,” in International conference on smart card research and advanced applications. Springer, 2010, pp. 182–193

    2010

  19. [27]

    Mining hardware assertions with guidance from static analysis,

    Hertz, Samuel et al. , “Mining hardware assertions with guidance from static analysis,” IEEE Transactions on Computer-Aided Design of Inte- grated Circuits and Systems , vol. 32, no. 6, pp. 952–965, 2013

    2013

  20. [28]

    Automap: Automated mapping of security properties between different levels of abstraction in design flow,

    Ahmed, Bulbul et al. , “Automap: Automated mapping of security properties between different levels of abstraction in design flow,” in 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD). IEEE, 2021, pp. 1–9

    2021

  21. [29]

    Verigen: A large language model for verilog code generation,

    Thakur, Shailja et al. , “Verigen: A large language model for verilog code generation,” ACM Transactions on Design Automation of Electronic Systems, vol. 29, no. 3, pp. 1–31, 2024

    2024

  22. [30]

    Asleep at the keyboard? assessing the security of github copilot’s code contributions,

    Pearce, Hammond et al., “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 754–768

    2022

  23. [31]

    Attend first, consolidate later: On the importance of attention in different llm layers,

    Ben-Artzy et al. , “Attend first, consolidate later: On the importance of attention in different llm layers,” arXiv preprint arXiv:2409.03621, 2024

    2024 arXiv

  24. [32]

    Full parameter fine-tuning for large language models with limited resources,

    Lv, Kai et al. , “Full parameter fine-tuning for large language models with limited resources,” arXiv preprint arXiv:2306.09782 , 2023

    2023 arXiv

  25. [33]

    Understanding the performance and estimating the cost of llm fine-tuning,

    Xia, Yuchen et al., “Understanding the performance and estimating the cost of llm fine-tuning,” in 2024 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2024, pp. 210–223

    2024

  26. [34]

    Data-efficient fine-tuning for llm-based recommenda- tion,

    Lin, Xinyu et al., “Data-efficient fine-tuning for llm-based recommenda- tion,” in Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval , 2024, pp. 365–374

    2024

  27. [35]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    White, Jules et al. , “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382 , 2023

    2023 arXiv

  28. [36]

    Trust-hub

    “Trust-hub.” [Online]. Available: https://trust-hub.org/#/home

  29. [37]

    Common weakness enumeration

    “Common weakness enumeration.” [Online]. Available: https://cwe. mitre.org/

  30. [38]

    Gpt-4 can’t reason,

    K. Arkoudas, “Gpt-4 can’t reason,” arXiv preprint arXiv:2308.03762 , 2023

    2023 arXiv

  31. [39]

    A mathematical investigation of hallucination and creativity in gpt models,

    M. Lee, “A mathematical investigation of hallucination and creativity in gpt models,” Mathematics, vol. 11, no. 10, p. 2320, 2023

    2023

  32. [40]

    A survey on large language model based autonomous agents,

    Wang, Lei et al., “A survey on large language model based autonomous agents,” Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    2024

  33. [41]

    Llm-powered autonomous agents,

    L. Weng, “Llm-powered autonomous agents,” lilianweng. github. io, Jun, vol. 23, 2023

    2023

  34. [42]

    Pyverilog: A python-based hardware design processing toolkit for verilog hdl,

    S. Takamaeda-Yamazaki, “Pyverilog: A python-based hardware design processing toolkit for verilog hdl,” in Applied Reconfigurable Comput- ing: 11th International Symposium, ARC 2015, Bochum, Germany, April 13-17, 2015, Proceedings 11 . Springer, 2015, pp. 451–460

    2015

  35. [43]

    Empowering hardware security with llm: The development of a vulnerable hardware database,

    Saha, Dipayan et al. , “Empowering hardware security with llm: The development of a vulnerable hardware database,” in 2024 IEEE Inter- national Symposium on Hardware Oriented Security and Trust (HOST) . IEEE, 2024, pp. 233–243

    2024

  36. [44]

    From rtl to sva: Llm-assisted generation of formal verification testbenches,

    Orenes-Vera, Marcelo et al., “From rtl to sva: Llm-assisted generation of formal verification testbenches,” arXiv preprint arXiv:2309.09437, 2023

    2023 arXiv

  37. [45]

    Chiraag: Chatgpt informed rapid and automated assertion generation,

    Mali, Bhabesh et al., “Chiraag: Chatgpt informed rapid and automated assertion generation,” in 2024 IEEE Computer Society Annual Sympo- sium on VLSI (ISVLSI) . IEEE, 2024, pp. 680–683

    2024

  38. [46]

    Gpt-4 technical report. arxiv 2303.08774,

    R. OpenAI, “Gpt-4 technical report. arxiv 2303.08774,” View in Article, vol. 2, no. 5, 2023

    2023 arXiv

  39. [47]

    Gemini: a family of highly capable multimodal models,

    Team, Gemini et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  40. [48]

    Claude 3 haiku: our fastest model yet,

    Anthropic, “Claude 3 haiku: our fastest model yet,” 2024. [Online]. Available: https://www.anthropic.com/news/claude-3-haiku

    2024

  41. [49]

    Meta llama 3.1

    M. AI, “Meta llama 3.1.” 2024. [Online]. Available: https://ai.meta. com/blog/meta-llama-3-1/

    2024

  42. [50]

    Search microsoft copilot: Your everyday ai companion

    Microsoft, “Search microsoft copilot: Your everyday ai companion.” [Online]. Available: https://copilot.microsoft.com/

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.