Pith. sign in

REVIEW 3 major objections 5 minor 39 references

HardSecBench, a 924-task benchmark spanning Verilog RTL and firmware C, aims to show that LLMs routinely generate hardware code that passes functional tests yet leaves security protections unimplemented, and that the security knowledge is p

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:23 UTC pith:72CC5TMH

load-bearing objection A genuinely useful benchmark with a load-bearing assumption about spec leakage that the paper never validates — worth refereeing, but the headline claim needs reframing. the 3 major comments →

arxiv 2601.13864 v2 pith:72CC5TMH submitted 2026-01-20 cs.CR cs.AI

HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation

Qirui Chen , Jingxian Shuai , Shuangwu Chen , Shenghao Ye , Zijian Wen , Xufei Su , Jie Jin , Jiangming Li
show 3 more authors
Jun Chen Xiaobin Tan Jian Yang
This is my paper
classification cs.CR cs.AI
keywords hardware securityLLM code generationbenchmarkVerilog RTLfirmware CCommon Weakness Enumerationsecurity awarenessprompt sensitivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces HardSecBench, a benchmark of 924 tasks across Verilog RTL and firmware C that evaluates whether LLMs produce hardware code satisfying security requirements when the specification mentions only functional behavior. Across current models, functional pass rates exceed 85% for most models while security pass rates sit around 30–45%, showing that functionally valid code can still carry exploitable flaws such as bypassable write-once locks. The authors find the security knowledge is largely present but dormant: adding a generic security reminder raises security pass rates, and naming the vulnerability class raises them further for strong models. The benchmark's evaluation is grounded in executable, requirement-level test harnesses rather than human judgment, with coverage and mutation gates to ensure the tests detect insecure implementations. The load-bearing premise is that the functional requirements do not reveal security intent, which the paper does not explicitly validate.

Core claim

On HardSecBench's terms, the central discovery is that LLMs routinely satisfy functional requirements while leaving the security protections unimplemented, and that the gap is not primarily a knowledge gap: when prompted with generic security hints or explicit vulnerability classes, stronger models' security pass rates jump substantially, indicating that security expertise is encoded in the model but not activated by purely functional prompts. The benchmark is built so that functional requirements deliberately conceal security intent, and evaluation uses simulation evidence from harnesses designed to trigger security-relevant behaviors.

What carries the argument

The carrying mechanism is a multi-agent construction pipeline that separates specification synthesis, golden-implementation synthesis, and test-harness synthesis into independent branches, then reconciles them through an arbiter-driven refinement loop. Each task carries a structured specification with functional requirements Rf and security requirements Rs; target models see only Rf while evaluation checks Rs via requirement-level harnesses that emit deterministic PASS/FAIL. A final quality gate applies code-coverage and mutation-analysis filtering so retained harnesses distinguish secure from insecure implementations.

Load-bearing premise

The benchmark's headline conclusion assumes that the functional requirements Rf do not themselves reveal security intent; if a specification such as 'write-once lock register' implicitly tells the model the lock must apply to every write path, then failing to guard the DMA path is a functional mistake, and the benchmark measures instruction-following rather than security intuition—the paper reports no explicit leakage check beyond a manual audit of 100 tasks.

What would settle it

A leakage test that presents Rf alone to independent human raters and asks them to infer which security checks will be evaluated; if raters reliably infer the protected behaviors, then the specification is leaking security intent and the security-vs-functional gap on HardSecBench would partly dissolve. Alternatively, a probe experiment that adds a functionally irrelevant clarification (e.g., 'DMA writes must not affect the lock bit') to Rf and observes whether security pass rates jump to near-functional levels.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Hardware teams cannot rely on functional test suites as a proxy for security, since high functional pass rates coexist with low security pass rates in the same generated code.
  • Security-aware prompting is a cheap intervention: generic security reminders raise security pass rates for many models, and explicit vulnerability hints raise them further, suggesting prompt design alone can steer safer code.
  • Functional refinement loops fix functional bugs but do not fix security flaws, so security must be checked explicitly rather than assumed to follow from correctness.
  • The weak correlation between security pass rate and general coding ability at baseline, and strong correlation under explicit hints, suggests that security failures for stronger models reflect failure to activate knowledge rather than lack of capability.
  • Domain-specific fine-tuning improves baseline security but the benefit depends on base-model strength, with some fine-tuned families degrading when security hints become highly explicit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same functional-vs-security gap likely appears in other high-stakes code-generation settings (e.g., cryptographic protocol code or firmware for physical devices); HardSecBench's harness methodology could be ported to those domains.
  • The dormant-knowledge result implies a practical deployment recipe: pair a code-generation LLM with a security-aware reranker or self-critique pass that is explicitly given the security requirements, and measure the Hint 0 vs Hint 2 gap to calibrate how much guidance a given model needs.
  • Because the benchmark's construction is itself LLM-driven, a cross-generator replication—building a second version of HardSecBench using a different construction LLM—would clarify whether the measured security gap is a property of target models or an artifact of the construction distribution.
  • A testable extension is to measure whether the security gap shrinks when the functional specification is made artificially unambiguous about protected interfaces; if it does, the benchmark conflates ambiguous instruction-following with missing security awareness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. HardSecBench is a benchmark of 924 Verilog RTL and firmware-C tasks across 76 hardware-relevant CWEs, designed to measure whether LLMs generate code that is functionally correct but lacks security protections. The benchmark is constructed by a multi-agent pipeline: an Architect separates each task into functional requirements Rf and security requirements Rs, an Expert generates a golden implementation, a Tester generates requirement-level executable harnesses, and an Arbiter reconciles mismatches. A final quality gate applies coverage and mutation filtering, supplemented by a 100-task manual audit. The paper evaluates 16 models in single-attempt and iterative-refinement settings, reporting high functional pass rates (mostly >85%) but much lower security pass rates (roughly 30–45%). A prompt-sensitivity study shows security scores rise when models are given general or specific security hints, which the authors interpret as evidence that security knowledge is present but dormant. The central claim is that models often satisfy stated functional specifications while failing to implement protections that would be required only if the model anticipated security intent.

Significance. If the benchmark's core validity assumptions hold, this is a useful and timely contribution. The paper ships a large, open benchmark with execution-grounded evaluation, requirement-level harnesses, and a quality gate that uses coverage and mutation analysis to filter weak tests. The design of decoupled artifact synthesis is a reasonable attempt to reduce implementation–verification coupling, and the prompt-sensitivity findings are a concrete, falsifiable result about how security performance responds to explicit guidance. The finding that security pass rates do not track general coding ability under no-hint conditions is also potentially important for future secure-code-generation research. However, the benchmark's value as a measure of latent security awareness depends critically on the assertion that Rf does not leak security intent, and that assertion is not validated in the manuscript. The use of the same LLM family to generate the benchmark artifacts and then to appear among the evaluated models is an additional external-validity concern. These issues are addressable, but they must be fixed before the results can be accepted as a reliable measure of security awareness.

major comments (3)
  1. [§3.1, §3.4, and Figure 1] The central claim that models are functionally correct but security-unaware depends on the assertion in §3.1 that Rf 'states only functional behavior and avoids revealing security intent.' No leakage audit is reported, and the manual audit in §3.4 is not described as checking this property. The motivating example in Figure 1(a) itself contains 'Lock mechanism after first CPU write' in the design request; if similar phrases appear in HardSecBench Rf entries, then a model failing to lock the DMA path is violating an explicit functional requirement, and the benchmark is measuring instruction-following rather than latent security intuition. Additionally, the Arbiter-driven refinement loop (§3.3) can revise Rf using full knowledge of Rs, creating a concrete mechanism for silent leakage. The authors should provide a leakage audit on the released tasks, e.g., keyword and human-annotation checks
  2. [§5 and Table 1] All benchmark artifacts were generated by Gemini-3-Pro-Preview (Architect, Expert, Tester, Arbiter), and the same exact model is then among the evaluated models in Table 1. This introduces a potential bias: the specifications, golden implementations, and harnesses may be style-matched to Gemini's outputs, inflating its security scores or otherwise making the benchmark less neutral for other models. This is not formal circularity, but it weakens the claim of an objective, reliable evaluation. The authors should either (a) construct and report results on an independent, expert-written holdout subset, or (b) provide a residual analysis showing that Gemini's HardSecBench scores are not systematically higher than predicted from its SWE-bench/VerilogEval rankings.
  3. [§4.1 and Table 1] In the single-attempt setting, security is measured on the first compilable implementation without requiring functional correctness. For models with low functional pass rates in Table 1 (e.g., Qwen3-Coder-30B, Llama-4-Scout), a security test can fail simply because the design is non-functional (e.g., a missing DMA write path), not because of a security-awareness failure. The iterative-refinement setting partially addresses this, but the 1-attempt@1 column is prominently reported and used to support the abstract's gap claim. The authors should report security pass rates conditioned on functional pass (e.g., Sec@1 given Func@1), or explicitly state that the single-attempt column is a joint functional/security measure. Otherwise the size of the reported functional-vs-security gap is confounded for weaker models.
minor comments (5)
  1. [Table 1 caption and §4.1] The table columns 'HSBench@1' and 'HSBench@5' are not explicitly labeled as the iterative-refinement setting. Please define these in the caption and clarify how many refinement iterations and how many samples n are used for Pass@5.
  2. [§6.3] The exact prompts for Hint 0, Hint 1, and Hint 2 are not given. Since the prompt-sensitivity analysis is a key contribution, the prompts should be included in an appendix or provided with the released code for reproducibility.
  3. [§5] The SWE-bench column in Table 1 appears to mix previously published scores with locally run evaluations, and some entries are missing. Specify the source for each value and explain missing entries (e.g., o1, Kimi).
  4. [§6.1] The Verilog coverage estimation via 'static analysis by mapping test harness signals to executable lines' is vague. Please describe the method concretely or state why simulation-based coverage is not feasible with Icarus Verilog.
  5. [General] Minor typos and wording issues: the abstract says 'may embed security flaws which could trigger induce catastrophic damages' and 'could trigger induce' is ungrammatical. Also, 'Data and code will be released soon' in the full text conflicts with the abstract's claim that they are available at the GitHub URL; align these statements.

Circularity Check

2 steps flagged

Security-awareness gap partly reduces to the unvalidated R_f/R_s separation; the paper's own example shows the lock policy inside the design request.

specific steps
  1. self definitional [Section 3.1 and Figure 1]
    "By design, R_f^i states only functional behavior and avoids revealing security intent, while R_s^i lists the protections required by the CWE guidance. ... during evaluation, we give the target model without R_s^i. [Figure 1(a):] Design a configuration register module with: CPU write interface / DMA write interface / Lock mechanism after first CPU write"

    The benchmark's headline claim—models satisfy functional requirements but lack security awareness—is operationally defined as failing R_s checks when only R_f is supplied. That interpretation is valid only if R_f never reveals security intent. Yet the paper's own illustrative design request explicitly includes 'Lock mechanism after first CPU write,' which is the write-once protection (CWE-1224) that the security harness enforces on the DMA path. If such phrasing occurs in the 924 generated R_f sets, then the Func >> Sec gap is partly a measure of whether the model followed an explicit lock requirement, not whether it inferred unstated security intent. The paper reports no leakage audit of R_f, and the Arbiter refinement loop (Section 3.3) can rewrite P_i using full knowledge of R_s, provid

  2. other [Section 3.3 and Section 3.4]
    "The Arbiter observes the structured specification P_i, the golden implementation, the requirement-level harnesses, and runtime evidence ... It then issues targeted feedback that specifies what to revise. ... domain experts manually audited 100 randomly sampled tasks and found that the artifacts largely satisfy HardSecBench's quality criteria."

    The only external validation (manual audit of 100 tasks) is reported as checking 'quality criteria' without stating that those criteria include the security-intent-free property of R_f. Meanwhile, the Arbiter-driven refinement loop has access to R_s and observes security-check failures, so it can modify the specification in ways that move security protections into R_f. Absent a post-hoc check that R_f remained free of security content, the central 'security awareness' measurement is not fully decoupled from the functional specification it is meant to be contrasted with.

full rationale

HardSecBench is an empirical benchmark rather than a mathematical derivation, so most equation-level circularity patterns do not apply. The Pass@k metric (Eq. 1) is standard and self-contained; construction and evaluation are decoupled in the sense that different agents write implementations and harnesses, and the quality gates (coverage, mutation, simulation) give independent evidence that the harnesses detect insecure implementations. However, the central inference—that a Func >> Sec gap indicates weak security awareness rather than explicit functional noncompliance—rests entirely on the Section 3.1 assertion that R_f 'states only functional behavior and avoids revealing security intent.' This is the benchmark's load-bearing premise, and the paper provides no validation of it. The contradiction in Figure 1 (the lock mechanism appears in the design request) and the Arbiter refinement mechanism (which can revise R_f using R_s) are concrete ways security intent can enter R_f. If that happens, the security pass rate is not a measure of unstated security awareness but of how well the model follows a stated lock policy, and the prompt-study finding that explicit hints activate 'dormant' knowledge is partly an artifact of explicitness in the specification. A leakage audit of R_f, or a manual check that the 100 audited tasks preserved the R_f/R_s separation, would resolve this. Because the issue affects the central claim but is a conditional construction risk rather than a proven identity, the score is 6 rather than higher. No self-citation or uniqueness-theorem circularity is present.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The benchmark's validity rests on hand-picked quality thresholds (80% coverage, 50% mutation detection) and on several domain assumptions about the separation of functional and security requirements, the correctness of LLM-generated golden references, and the fidelity of functional simulation for security properties. No novel physical entities are introduced.

free parameters (6)
  • Coverage threshold = 80%
    Hand-chosen quality gate for retaining benchmark tasks; affects which tasks enter the final 924.
  • Mutation detection threshold = 50%
    Hand-chosen minimum mutation score for harness retention; tasks below are filtered.
  • Number of mutants per task = 5
    Five mutant operators (constant change, operator swap, condition negation, stuck-at signal, assignment removal); a small hand-picked set.
  • Iteration cap for refinement = 5
    Evaluation refinement loop capped at 5 iterations; affects measured performance.
  • Compile-fix rounds = 3
    Single-attempt setting allows up to 3 rounds of compiler-error-only repair; affects functional/security pass rates.
  • Collaborator temperature = 0.3
    Set to keep functional feedback stable; affects refinement outcomes.
axioms (4)
  • domain assumption The functional requirements Rf do not reveal security intent
    Section 3.1 states this 'by design' but no leakage test or manual validation is reported. If Rf contains security hints, the benchmark measures instruction-following, not security awareness.
  • domain assumption The golden implementation is secure and correct
    The Expert agent generates it from Rf+Rs; correctness only checked against LLM-generated harnesses and a 100-task manual audit (Section 3.4).
  • domain assumption Functional simulation with Icarus Verilog/gcc faithfully exposes all security-relevant behaviors for 76 CWEs
    Section 5 uses Icarus Verilog (functional simulation) and gcc. Timing/clock/power-related CWEs (e.g., CWE-1206) may not be detectable in functional simulation, undermining coverage claims for those categories.
  • domain assumption CWE definitions map one-to-one to testable atomic requirements
    Sections 3.1-3.2 assume the Seed Generator and Architect convert CWE descriptions into requirements with valid one-to-one harnesses; no independent expert verification beyond a 100-sample audit.

pith-pipeline@v1.3.0-alltime-deepseek · 12941 in / 13739 out tokens · 133431 ms · 2026-08-03T09:23:55.352800+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used for hardware and firmware code generation, but existing studies primarily evaluate functional correctness while largely overlooking security. However, LLM-generated code that appears functionally sound may embed security flaws which could induce catastrophic damages after deployment. This critical research gap motivates us to design a benchmark for assessing security awareness under realistic specifications. In this work, we introduce HardSecBench, a benchmark with 924 tasks spanning Verilog Register Transfer Level (RTL) and firmware-level C, covering 76 hardware-relevant Common Weakness Enumeration (CWE) entries. Each task includes a structured specification, a secure reference implementation, and executable tests. To automate artifact synthesis, we propose a multi-agent pipeline that decouples synthesis from verification and grounds evaluation in execution evidence, enabling reliable evaluation. We evaluate diverse LLMs and find that they often satisfy functional requirements while leaving security risks. We also find that security results vary with prompting. These findings highlight pressing challenges and offer actionable insights for future advancements in LLM-assisted hardware design. Our data and code are available at https://github.com/chenqirui2002/HardSecBench.

Figures

Figures reproduced from arXiv: 2601.13864 by Jiangming Li, Jian Yang, Jie Jin, Jingxian Shuai, Jun Chen, Qirui Chen, Shenghao Ye, Shuangwu Chen, Xiaobin Tan, Xufei Su, Zijian Wen.

Figure 1
Figure 1. Figure 1: Potential security vulnerability in LLM-generated code. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: HardSecBench construction pipeline. From CWE-derived seeds, the Architect builds a structured specification [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: HardSecBench evaluation architecture. The target model [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Quality validation of HardSecBench: (a) Code coverage [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Security pass rates across different security hint levels for [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Correlation between code generation capability and secu [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of functional and security performance for [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: CWE pass rates for different model categories: (a) closed [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 1 linked inside Pith

  1. [1]

    Toward hardware security benchmarking of llms

    [Afsharmazayejaniet al., 2024 ] Raheel Afsharmazayejani, Mohammad Moradi Shahmiri, Parker Link, Hammond Pearce, and Benjamin Tan. Toward hardware security benchmarking of llms. In2024 IEEE LLM Aided Design Workshop (LAD), pages 1–5. IEEE,

    2024

  2. [4]

    Chip-Chat: Chal- lenges and opportunities in conversational hardware de- sign

    [Blockloveet al., 2023 ] Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. Chip-Chat: Chal- lenges and opportunities in conversational hardware de- sign. In2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD), pages 1–6,

    2023

  3. [6]

    SecV: Llm-based secure verilog generation with clue- guided exploration on hardware-cwe knowledge graph

    [Fanet al., 2025 ] Fanghao Fan, YingJie Xia, and Li Kuang. SecV: Llm-based secure verilog generation with clue- guided exploration on hardware-cwe knowledge graph. In James Kwok, editor,Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 8049–8057. International Joint Confer- ences on Artificial Inte...

    2025

  4. [7]

    [Fanget al., 2024 ] Wenji Fang, Mengming Li, Min Li, Zhiyuan Yan, Shang Liu, Hongce Zhang, and Zhiyao Xie

    Main Track. [Fanget al., 2024 ] Wenji Fang, Mengming Li, Min Li, Zhiyuan Yan, Shang Liu, Hongce Zhang, and Zhiyao Xie. AssertLLM: Generating hardware verification assertions from design specifications via multi-llms. In2024 IEEE LLM Aided Design Workshop (LAD), pages 1–1,

    2024

  5. [8]

    Constrained decoding for secure code gen- eration.arXiv preprint arXiv:2405.00218,

    [Fuet al., 2024 ] Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. Constrained decoding for secure code gen- eration.arXiv preprint arXiv:2405.00218,

    Pith/arXiv arXiv 2024

  6. [9]

    Model cards Simple, structured overviews of how an advanced AI model was designed and evaluated.,

    [Google, 2025] Google. Model cards Simple, structured overviews of how an advanced AI model was designed and evaluated.,

    2025

  7. [10]

    SecFSM: Knowledge graph-guided ver- ilog code generation for secure finite state machines in systems-on-chip,

    [Huet al., 2025 ] Ziteng Hu, Yingjie Xia, Xiyuan Chen, and Li Kuang. SecFSM: Knowledge graph-guided ver- ilog code generation for secure finite state machines in systems-on-chip,

    2025

  8. [11]

    SWE-bench: Can language mod- els resolve real-world github issues? InThe Twelfth Inter- national Conference on Learning Representations,

    [Jimenezet al., 2024 ] Carlos E Jimenez, John Yang, Alexan- der Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language mod- els resolve real-world github issues? InThe Twelfth Inter- national Conference on Learning Representations,

    2024

  9. [12]

    Rethinking safety in LLM fine-tuning: An optimization perspective

    [Kimet al., 2025 ] Minseon Kim, Jin Myung Kwak, Lama Alssum, Bernard Ghanem, Philip Torr, David Krueger, Fazl Barez, and Adel Bibi. Rethinking safety in LLM fine-tuning: An optimization perspective. InSecond Con- ference on Language Modeling,

    2025

  10. [13]

    SafeGenBench: A benchmark framework for security vul- nerability detection in llm-generated code,

    [Liet al., 2025 ] Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu. SafeGenBench: A benchmark framework for security vul- nerability detection in llm-generated code,

    2025

  11. [14]

    VerilogEval: Evaluating large language models for verilog code generation

    [Liuet al., 2023 ] Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. VerilogEval: Evaluating large language models for verilog code generation. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pages 1–8,

    2023

  12. [15]

    ChipNeMo: Domain-adapted llms for chip design,

    [Liuet al., 2024 ] Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catan- zaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, S...

    2024

  13. [16]

    [Liuet al., 2025 ] Shang Liu, Wenji Fang, Yao Lu, Jing Wang, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. RTLCoder: Fully open-source and efficient llm-assisted rtl code generation technique.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Sys- tems, 44(4):1448–1461,

    2025

  14. [17]

    VerilogLA VD: Llm-aided rule generation for vulnerability detection in verilog,

    [Longet al., 2025 ] Xiang Long, Yingjie Xia, Xiyuan Chen, and Li Kuang. VerilogLA VD: Llm-aided rule generation for vulnerability detection in verilog,

    2025

  15. [18]

    RTLLM: An open-source benchmark for de- sign rtl generation with large language model

    [Luet al., 2024 ] Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. RTLLM: An open-source benchmark for de- sign rtl generation with large language model. In2024 29th Asia and South Pacific Design Automation Confer- ence (ASP-DAC), pages 722–727. IEEE,

    2024

  16. [19]

    CircuitGuard: Mitigating llm memorization in rtl code generation against ip leakage

    [Mashnooret al., 2025 ] Nowfel Mashnoor, Mohammad Akyash, Hadi Kamali, and Kimia Azar. CircuitGuard: Mitigating llm memorization in rtl code generation against ip leakage. In2025 IEEE 43rd International Conference on Computer Design (ICCD), pages 790–797,

    2025

  17. [20]

    Llama 4 Leading Intelligence

    [Meta, 2025] Meta. Llama 4 Leading Intelligence. Unrivaled speed and efficiency., April

    2025

  18. [21]

    Common Weakness Enumeration, January

    [MITRE, 2006] MITRE. Common Weakness Enumeration, January

    2006

  19. [23]

    Introducing OpenAI o1-preview, December

    [OpenAI, 2024] OpenAI. Introducing OpenAI o1-preview, December

    2024

  20. [24]

    OpenRouter: The Unified Interface For LLMs,

    [OpenRouter, Inc, 2023] OpenRouter, Inc. OpenRouter: The Unified Interface For LLMs,

    2023

  21. [25]

    Au- toSV A: Democratizing formal verification of RTL module interactions

    [Orenes-Veraet al., 2021 ] Marcelo Orenes-Vera, Aninda Manocha, David Wentzlaff, and Margaret Martonosi. Au- toSV A: Democratizing formal verification of RTL module interactions. In58th ACM/IEEE Design Automation Con- ference, DAC 2021, San Francisco, CA, USA, December 5-9, 2021, pages 535–540. IEEE,

    2021

  22. [27]

    [Perryet al., 2023 ] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with ai assistants? InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS ’23, page 2785–2799, New York, NY , USA,

    2023

  23. [28]

    [Pinckneyet al., 2025 ] Nathaniel Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, and Brucek Khailany

    Association for Computing Machinery. [Pinckneyet al., 2025 ] Nathaniel Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, and Brucek Khailany. Revisiting VerilogEval: A year of improvements in large-language models for hardware code generation. ACM Trans. Des. Autom. Electron. Syst., 30(6), October

    2025

  24. [29]

    HDLCoRe: A training-free framework for miti- gating hallucinations in llm-generated hdl

    [Pinget al., 2025 ] Heng Ping, Shixuan Li, Peiyu Zhang, Anzhe Cheng, Shukai Duan, Nikos Kanakaris, Xiongye Xiao, Wei Yang, Shahin Nazarian, Andrei Irimia, and Paul Bogdan. HDLCoRe: A training-free framework for miti- gating hallucinations in llm-generated hdl. In2025 IEEE International Conference on LLM-Aided Design (ICLAD), pages 108–116,

    2025

  25. [30]

    AssertionBench: A benchmark to evaluate large-language models for asser- tion generation

    [Pulavarthiet al., 2025 ] Vaishnavi Pulavarthi, Deeksha Nan- dal, Soham Dan, and Debjit Pal. AssertionBench: A benchmark to evaluate large-language models for asser- tion generation. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Findings of the Association for Com- putational Linguistics: NAACL 2025, pages 8058–8065, Albuquerque, New Mexico, April

    2025

  26. [31]

    [Qiuet al., 2024 ] Ruidi Qiu, Grace Li Zhang, Rolf Drech- sler, Ulf Schlichtmann, and Bing Li

    Association for Computational Linguistics. [Qiuet al., 2024 ] Ruidi Qiu, Grace Li Zhang, Rolf Drech- sler, Ulf Schlichtmann, and Bing Li. AutoBench: Auto- matic testbench generation and evaluation using llms for hdl design. InProceedings of the 2024 ACM/IEEE Inter- national Symposium on Machine Learning for CAD, ML- CAD ’24, New York, NY , USA,

    2024

  27. [32]

    RES- CUE: Retrieval augmented secure code generation,

    [Shi and Zhang, 2025] Jiahao Shi and Tianyi Zhang. RES- CUE: Retrieval augmented secure code generation,

    2025

  28. [33]

    SiliconFlow: One Plat- form, All Your AI Inference Needs, August

    [SiliconFlow, 2023] SiliconFlow. SiliconFlow: One Plat- form, All Your AI Inference Needs, August

    2023

  29. [34]

    SpyGlass Static and Formal Verification,

    [Synopsys, 2015] Synopsys. SpyGlass Static and Formal Verification,

    2015

  30. [36]

    ChatEDA: A large language model powered autonomous agent for eda.IEEE Transactions on Computer-Aided De- sign of Integrated Circuits and Systems,

    [Wuet al., 2024 ] Haoyuan Wu, Zhuolun He, Xinyun Zhang, Xufeng Yao, Su Zheng, Haisheng Zheng, and Bei Yu. ChatEDA: A large language model powered autonomous agent for eda.IEEE Transactions on Computer-Aided De- sign of Integrated Circuits and Systems,

    2024

  31. [37]

    [Yuet al., 2025b ] Zhongzhi Yu, Mingjie Liu, Michael Zim- mer, Yingyan Celine, Yong Liu, and Haoxing Ren

    Association for Computing Machinery. [Yuet al., 2025b ] Zhongzhi Yu, Mingjie Liu, Michael Zim- mer, Yingyan Celine, Yong Liu, and Haoxing Ren. Spec2rtl-agent: Automated hardware code generation from complex specifications using llm agent systems. In 2025 IEEE International Conference on LLM-Aided De- sign (ICLAD), pages 37–43,

    2025

  32. [38]

    GLM-4.6: Advanced Agentic, Reasoning and Coding Capabilities, September

    [Z.ai, 2025] Z.ai. GLM-4.6: Advanced Agentic, Reasoning and Coding Capabilities, September

    2025

  33. [39]

    QiMeng-CodeV-R1: Reasoning-enhanced verilog genera- tion

    [Zhuet al., 2025 ] Yaoyu Zhu, Di Huang, Hanqi Lyu, Xi- aoyun Zhang, Chongxiao Li, Wenxuan Shi, Yutong Wu, Jianan Mu, Jinghua Wang, Yang Zhao, Pengwei Jin, Shuyao Cheng, Shengwen Liang, Xishan Zhang, Rui Zhang, Zidong Du, Qi Guo, Xing Hu, and Yunji Chen. QiMeng-CodeV-R1: Reasoning-enhanced verilog genera- tion. InAdvances in Neural Information Processing S...

    2025

  34. [2006]

    Kimi K2: Open Agentic Intelligence, July

    [Moonshot AI, 2025] Moonshot AI. Kimi K2: Open Agentic Intelligence, July

    2025

  35. [2021]

    CWEval: Outcome-driven evaluation on functionality and security of llm code gen- eration

    [Penget al., 2025 ] Jinjun Peng, Leyi Cui, Kele Huang, Jun- feng Yang, and Baishakhi Ray. CWEval: Outcome-driven evaluation on functionality and security of llm code gen- eration. In2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 33– 40,

    2025

  36. [2023]

    DeepSeek-V3 techni- cal report,

    [DeepSeek-AI, 2024] DeepSeek-AI. DeepSeek-V3 techni- cal report,

    2024

  37. [2024]

    Claude Model cards, Novem- ber

    [Anthropic, 2025] Anthropic. Claude Model cards, Novem- ber

    2025

  38. [2025]

    CyberSecEval 2: A wide-ranging cyberse- curity evaluation suite for large language models,

    [Bhattet al., 2024 ] Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. CyberSecEval 2: A wide-ranging cyberse- curity evaluation suite for large language models,

    2024

  39. [2026]

    Bench- marking large language models for automated verilog rtl code generation

    [Thakuret al., 2023 ] Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. Bench- marking large language models for automated verilog rtl code generation. In2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1–6,

    2023