{"id":"3d39b78f-ba46-44eb-9fc7-01f6530b51f2","arxiv_id":"2504.21770","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LASHED pairs LLMs with linting and formal assertions to flag hardware security bugs in RTL, reaching 87.5% precision on its recommended prompt scheme.","lead":"This paper combines large language models with static analysis tools to find security bugs in chip designs, testing the combination on four open-source RISC-V system-on-chip designs across five hardware vulnerability classes. It reports that 87.5% of the issues flagged by its recommended setup are plausible vulnerabilities, and that adding examples and asking the model to reconsider improves precision.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"87.5% precision rests on author-confirmed labels with no independent ground truth for three of four SoCs, amplified by best-of-8 config selection and v3 in-context examples taken from evaluated H@DAC-21; as supported, the headline measures author-system agreement rather than validated detection.","rationale":"The reader's weakest assumption — that author-confirmed violations are valid ground truth — is exactly the load-bearing point, so I record agreement. The central claim is a precision estimate, and both the numerator (35 true positives) and the configuration choice (best of eight cells on the same data) rest on the authors' own judgment; only H@DAC-21 offers an external anchor, and even there no scoring against the known-bug list is reported. Credit is due: the pipeline is concrete, the arithmetic is internally consistent (Table III reconciles: 545 flagged, 277 TP, 50.8%; per-variation precisions 0.44/0.52/0.50/0.62; gpt-4o+v3 gives 35/40), FPV falsifications and lint hits are objective outputs, the prompt-variation analysis is a useful contribution, and the paper explicitly acknowledges the manual-evaluation limitation and the impossibility of recall (Sections IV and V). The concern is not honesty but validity: 'plausible CWE' as judged by the system designers, without inter-annotator checks, does not establish that 87.5% of flags are genuine weaknesses, and best-of-8 selection on the same data plus v1/v3 examples admitted to be 'inspired from the Hack@DAC-21 SoC' (Section IV-A2) make the number optimistic for transfer. I therefore keep the reader's CONDITIONAL verdict: the approach is promising and publishable, with the quantitative claim scoped as author-confirmed plausibility pending an external audit. The absence of released artifacts and of LLM-only or static-only baselines is a secondary gap that does not change the verdict.","tokens_in":14341,"tokens_out":19625,"duration_ms":186316,"concrete_test":"Blinded external re-labeling audit. Recruit two hardware-security experts unaffiliated with the paper. For each flagged instance in the recommended cell (40 instances; or all 545 for robustness), provide the RTL context, the raw static-analysis output (lint violation lines or falsified assertion), and the target CWE — but not LASHED's LLM explanation, the author TP/FP labels, or the system's name. Ask each expert to independently classify each instance as a plausible instance of the target CWE. Compute Cohen's kappa between the two experts and between each expert and the author labels, and recompute the gpt-4o+v3 precision under each expert's labels with a Wilson interval. Within the same audit, recompute precision on H@DAC-21 restricted to the published injected-bug list.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim — 87.5% of instances flagged by our recommended scheme are plausible CWEs (Section IV; 35/40 for gpt-4o+v3) — has a numerator built entirely from author judgment. Section IV states 'We evaluated the violations manually, with our author-confirmed violations providing the True Positives count', and Section V concedes 'manual evaluation of flagged violations' via 'visual inspection'. Although Section III-B says the five CWEs were chosen because they occur in H@DAC-21 'to have some ground-truth', no SoC is scored against a known-bug list in the reported results: the true positives are plausibility judgments for all four SoCs, with no second coder, no inter-annotator statistic, and no benchmark cross-check. Three compounding factors make the figure fragile even under good faith. (1) Selection: 87.5% is the maximum of eight configurations (2 LLMs × 4 variations), chosen after inspecting all results on the same data; aggregate precision is 277/545 = 50.8%, the best variation alone is 62%, and the gap on a cell of only 40 instances (Wilson 95% CI roughly [74%, 95%]) is too large to attribute to prompt design. The v3-vs-v1 gain (62% vs 52%) is itself not statistically significant at these counts (z ≈ 1.5). (2) Contamination: Section IV-A2 says v1's in-context examples are 'inspired from the Hack@DAC-21 SoC', one of the four evaluated SoCs; v3 includes v1, so the recommended prompt is partly tuned to an evaluated test item, and the paper calls H@DAC-21's 0.55 precision 'expected' for this reason. (3) The stated mitigation — 'each violation... is accompanied by a failing assertion or violation' — is weakest where precision is highest: for linter-based CWEs 1191 and 1300, the violation is a structural pattern (width mismatch, if-without-else), so the security-relevance judgment is the author label itself. As supported, 87.5% measures agreement between LASHED and its designers on the evaluation set, not a validated detection rate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LASHED, a three-stage pipeline for early detection of RTL hardware security bugs: an LLM identifies CWE-relevant assets; a static analysis stage (either lint checks or assertion-based formal verification, depending on the CWE) flags violations; and a second LLM pass prunes false positives and provides explanations. The authors evaluate LASHED on four open-source RISC-V SoCs across five CWEs, four prompt variations (v0-v3), and two LLMs (gpt-4o-mini and gpt-4o), reporting aggregate precision and a headline claim that the recommended configuration (gpt-4o with variation v3) achieves 87.5% precision (35 of 40 flagged instances). They also analyze the effect of in-context learning and self-reflection prompts on precision.","tokens_in":14696,"tokens_out":7002,"duration_ms":66408,"significance":"The direction is timely and potentially useful: combining static analysis with LLM contextualization for hardware security checking is under-explored, and the paper provides a concrete instantiation with explicit CWE-to-lint/assertion mappings and a multi-SoC study. If the headline precision were robust, this would be a meaningful step toward practical pre-silicon security screening. The paper also deserves credit for reporting the aggregate 50.8% precision and per-CWE breakdowns, which makes the variability across CWEs visible. However, the central quantitative claim is currently supported only by author-confirmed labels, post-hoc configuration selection, and a prompt that is partly derived from one of the evaluation SoCs, so the evidence is not yet convincing.","major_comments":[{"comment":"The 87.5% headline is the maximum of eight experimental configurations (2 LLMs x 4 prompt variations) and is selected after inspecting the results on the same data. The paper's own aggregate numbers are much lower: 277/545 = 50.8% overall, and the best average variation precision is 62% for v3. No confidence intervals or significance tests are reported; for the 40-flagged-instance cell behind the headline, the 95% Wilson interval spans roughly [74%, 95%], so the point estimate is compatible with substantially lower precision. The claim that in-context learning and 'thinking again' improve precision needs a pre-registered or held-out selection procedure and a statistical comparison.","section":"Section IV and Table III"},{"comment":"The paper states in Section IV-A2 that the v1 guiding examples are 'inspired from the Hack@DAC-21 SoC', and v3 includes v1, yet Hack@DAC-21 is one of the four evaluation SoCs. This is a test-set leakage for the recommended prompt. The contradiction with Section III-C, which says the v1 examples are taken from the MITRE website, must be resolved. Moreover, Section III-B says the CWEs were selected because they appear in Hack@DAC-21 'to have some ground-truth', but the reported precision never uses that ground truth: the paper does not check whether LASHED flags the known injected bugs. As a result, the evaluation is circular for one SoC and unvalidated for the other three.","section":"Section IV-A2, Section III-C, and Appendix B"},{"comment":"The true-positive counts are 'author-confirmed violations' established by visual inspection (Section V), with no second annotator, no inter-annotator agreement statistic, and no independent benchmark for OpenTitan, E203, or Veerwolf. The authors explicitly acknowledge this limitation, but the abstract and conclusion still present 87.5% as an empirical result. Since precision is the only performance metric reported, the headline should be reframed as 'author-validated precision' and corroborated by external labels or at minimum a blinded second annotation.","section":"Section IV and Section V"},{"comment":"Each experiment appears to be a single run with no control for LLM stochasticity (temperature, seed, or repeated calls), and some cells have very small counts (e.g., CWE 1244 flagged cells with 1-3 instances). The quantitative comparisons between prompt variations and between LLMs therefore rest on unstable point estimates; repeated runs or confidence intervals are needed before drawing conclusions.","section":"Section IV and Appendix A"}],"minor_comments":[{"comment":"Typos: 'Identificaiton' in Section II-A, 'to conduct out experiments' in Section III-D, 'gpt-40-mini' in Section IV-A3, 'Propmpt' in Appendix B, and 'time∝ loc0.53 approximately ... produces follows the relation' in Appendix A.","section":"Throughout"},{"comment":"Table III defines FDR as FP/Flagged, which is not the standard false discovery rate; use 'false positive proportion' or '1 - precision' instead.","section":"Table III"},{"comment":"Figures 4 and 5 use the label 'Variation(LLM)' and do not show numerical values on the stacked bars, making it difficult to verify the totals in Table III from the figures.","section":"Figures 4 and 5"},{"comment":"The code snippets in Section II (e.g., the assertion templates with 'd i s a b l e i f f') contain spacing artifacts from the PDF rendering; ensure these are formatted correctly in the final version.","section":"Section II-B"}],"recommendation":"major_revision","confidential_remarks":"The paper would benefit from an explicit artifact release, since the commercial tools (SpyGlass, VC Formal, Verific) are not open and no code or prompts are provided, which limits reproducibility. For a journal submission, I would ask the authors to make prompts and pipeline scripts available and to consider a known-bug benchmark (e.g., the H@DAC-21 injected bugs) as a primary evaluation before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the pipeline is genuinely new for RTL security, and the prompt engineering story is useful; but the headline 87.5% precision is not robustly supported. The ground truth is the authors' own labels for most SoCs, and the best configuration is selected from the same data. I'd send it to review, but the authors need to either release artifacts and get independent labels, or soften the central claim.\n\nThe combination of LLM with static analysis for hardware CWE detection is a real step beyond prior LLM-only works like SoCureLLM and Flag-RTL. The idea of using the LLM for asset identification, then linter or formal assertions, then LLM contextualization to prune false positives is coherent. The four prompt variations give concrete evidence about in-context learning and self-reflection, and the cost/scalability data is a nice practical touch.\n\nThe main issue is evaluation. Section IV says true positives are author-confirmed. For three of four SoCs there is no benchmark, so precision is measured against the authors' own plausibility judgments. Section IV-A2 notes the v1 in-context examples are 'inspired from' H@DAC-21, one of the four evaluated SoCs, and v3 inherits them. The 87.5% figure is the best of eight configurations selected after inspecting all results; aggregate precision is about 51%, and the v3-vs-v1 gap is not statistically significant at these counts. The linter-based CWEs (1191, 1300) are structural patterns, so the 'failing assertion or violation' does not independently establish security relevance. None of this means the approach is wrong, but as written the headline measures author-system agreement on the evaluation set, not a validated detection rate. No artifacts are released, which makes replication impossible.\n\nThis is for researchers working on LLM-assisted hardware verification. A serious referee could help turn this into a solid paper by demanding independent labels (e.g., known-bug benchmarks for at least some SoCs), reporting per-configuration statistics, and either releasing the prompts and code or providing a clear ablation.\n\nI would accept it for peer review and ask for major revision. The core idea deserves a place in the literature if the evaluation is cleaned up.","headline":"Useful pipeline and prompt analysis, but the 87.5% precision claim is not independently validated; send to review with a request for harder ground truth.","tokens_in":15403,"tokens_out":2372,"would_cite":true,"duration_ms":23115,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pipeline that pairs a large language model with static analysis can flag plausible hardware security bugs in RTL code with 87.5 percent precision.","keywords":["hardware security","RTL bug detection","static analysis","large language models","Common Weakness Enumeration","SystemVerilog assertions","RISC-V SoCs","in-context learning"],"falsifier":"Run the recommended LASHED configuration blind on a clean open-source RISC-V design into which known instances of CWE-1191, CWE-1231, CWE-1233, CWE-1244, and CWE-1300 have been injected, then compare its flagged locations against the injection list; if substantially fewer than 87.5% of its flags match independently known bugs, the reported precision depends on the authors' labeling rather than on the method.","tokens_in":14064,"feed_emoji":"🛡️","tokens_out":9147,"duration_ms":88393,"temperature":0.7,"pith_summary":"LASHED is a proposed three-step pipeline for detecting security bugs in register-transfer-level (RTL) hardware designs, combining a large language model with static analysis so that each covers the other's blind spots. The LLM first names security-relevant signals in the code; a linter or formal property checker then turns those signals into concrete checks; the LLM finally prunes false alarms and writes a plain-language explanation for each remaining issue. The paper's central result is that on four open-source RISC-V system-on-chips, across five hardware Common Weakness Enumerations, the recommended configuration (a stronger LLM with an in-context example and a \"think again\" step) flagged 40 instances, 35 of which the authors judged to be plausible CWEs, a precision of 87.5%. The authors argue this makes early, explainable security screening feasible without requiring a full formal-verification setup for every check.","feed_headline":"LLM plus static checks flag plausible chip bugs 87.5% of the time","feed_subtitle":"Best setup: 35 of 40 flags matched plausible security weaknesses across four open-source RISC-V chips.","key_machinery":"The engine of LASHED is the three-stage division of labor it calls AssetID, Static Analysis, and Contextualization. In AssetID, the LLM is given RTL source plus a hardware CWE description and asked to list security-relevant signals, such as lock bits, privilege levels, or debug-access passwords. Static Analysis then uses those signals to focus an existing tool: for structural weaknesses it runs lint checks on assignments and conditionals, and for behavioral weaknesses it fills custom SystemVerilog assertion templates—asserting that lock signals stay stable, that protected registers change only under proper lock, and that privilege escalation obeys its conditions—and checks them with formal property verification. Contextualization sends the lint violations and falsified assertions back to the LLM, which decides which ones are genuine security issues and explains them. The key mechanism is that the LLM supplies the semantic context static tools lack, while the static tools supply the verifiable evidence LLMs do not generate on their own.","core_discovery":"The central claim is that combining an LLM with static analysis detects real hardware security weaknesses in RTL code that neither tool reliably finds alone, and that the combination can be steered by prompt engineering. In the paper's evaluation, a total of 545 instances were flagged across 160 experiments, of which 51% were considered plausible CWEs; precision varied by weakness, from 100% for improper debug access control (CWE-1191) down to 28% for improper lock-bit protection (CWE-1231). The recommended scheme, gpt-4o with prompt variation v3, achieved 87.5% precision: 35 of 40 flagged instances. The paper also reports that in-context examples mainly improve asset identification, asking the model to \"think again\" mainly removes false positives, and the stronger model produces fewer false positives overall.","pith_inferences":["A natural extension beyond the paper's five CWEs would be to test whether the same two-template style scales to other hardware weakness classes; the paper itself notes that CWE-1233 was hardest because protected registers take many forms, so new templates would likely be needed.","A more decisive evaluation would use injected-bug benchmarks to measure recall, not just precision; the paper currently can only confirm bugs after flagging them, so it never measures how many real bugs were missed.","Given that the v1 examples were drawn from one of the evaluated SoCs, I would expect precision on a completely unrelated CWE-specific benchmark to be lower; measuring that drop would separate in-context learning from memorization."],"forward_implications":["For a given CWE and SoC, LASHED reports only a handful of candidate issues (3.4 on average), each with a source line and explanation, instead of drowning the designer in raw lint warnings.","Design teams can apply the same prompts to large, unfamiliar designs: the approach was run on a root-of-trust chip of about 171k lines of code, not just the small benchmark used for prototyping.","Prompt design is a first-class knob: adding one in-context example and a second look at contextualization raised precision from 44% to 62% across all runs, and 87.5% in the recommended configuration.","The two static-analysis styles work for different weakness families, so a practical deployment would choose linting for structural or side-channel checks and assertions for lock-bit and privilege checks."],"supporting_citations":[{"why":"supplies the benchmark SoC with known planted bugs used for initial validation and as the source of the in-context examples.","marker":"[30]"},{"why":"adds a large root-of-trust SoC to test whether the approach scales beyond the small prototyping benchmark.","marker":"[31]"},{"why":"one of the open-source RISC-V SoCs in the evaluation set, contributing flagged-instance counts.","marker":"[32]"},{"why":"one of the open-source RISC-V SoCs in the evaluation set, used for cross-SoC precision comparison.","marker":"[33]"},{"why":"provides the hardware CWE taxonomy and the coded examples embedded in prompt variation v1.","marker":"[27]"},{"why":"supplies the two LLM backends (gpt-4o-mini and gpt-4o) used for asset identification and contextualization.","marker":"[35]"},{"why":"is the linter whose tags generate the structural violations sent to contextualization for the lint-based CWEs.","marker":"[14]"},{"why":"is the formal property verification tool that checks the custom assertion templates and produces falsified properties.","marker":"[29]"}],"fun_headline_variants":["LLM + static analysis catches plausible chip bugs 87.5% of time","Hybrid LLM/static tool nails chip security bugs with 87.5% precision","LASHED: LLM paired with static checks finds real RTL bugs early","Ask LLM to 'think again' boosts chip bug detection to 87.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the authors' manual review of flagged violations is a valid ground truth; the headline precision comes from those labels and from a configuration chosen after inspecting the same results, so biased labeling or selection would overstate the claim.","fun_headline_variants_meta":{"raw":{"variants":["LLM + static analysis catches plausible chip bugs 87.5% of time","Hybrid LLM/static tool nails chip security bugs with 87.5% precision","LASHED: LLM paired with static checks finds real RTL bugs early","Ask LLM to 'think again' boosts chip bug detection to 87.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1285,"prompt_tokens":862,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":478,"tokens_out":423,"duration_ms":4382,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:54:00.762011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the recommended LASHED configuration blind on a clean open-source RISC-V design into which known instances of CWE-1191, CWE-1231, CWE-1233, CWE-1244, and CWE-1300 have been injected, then compare its flagged locations against the injection list; if substantially fewer than 87.5% of its flags match independently known bugs, the reported precision depends on the authors' labeling rather than on the method.","supporting_citations":[{"cited_title":"HACK-EVENT/hackatdac21,","cited_arxiv_id":null,"evidence_quote":"supplies the benchmark SoC with known planted bugs used for initial validation and as the source of the in-context examples."},{"cited_title":"Open source silicon root of trust (RoT) | OpenTitan,","cited_arxiv_id":null,"evidence_quote":"adds a large root-of-trust SoC to test whether the approach scales beyond the small prototyping benchmark."},{"cited_title":"Available: https://www.synopsys.com/verification/ static-and-formal-verification/vc-formal.html","cited_arxiv_id":null,"evidence_quote":"one of the open-source RISC-V SoCs in the evaluation set, contributing flagged-instance counts."},{"cited_title":"VeeRwolf,","cited_arxiv_id":null,"evidence_quote":"one of the open-source RISC-V SoCs in the evaluation set, used for cross-SoC precision comparison."},{"cited_title":"CWE - CWE Most Important Hardware Weaknesses,","cited_arxiv_id":null,"evidence_quote":"provides the hardware CWE taxonomy and the coded examples embedded in prompt variation v1."},{"cited_title":"Hummingbirdv2 E203 Core and SoC - GitHub,","cited_arxiv_id":null,"evidence_quote":"supplies the two LLM backends (gpt-4o-mini and gpt-4o) used for asset identification and contextualization."},{"cited_title":"Synopsys VC SpyGlass Lint,","cited_arxiv_id":null,"evidence_quote":"is the linter whose tags generate the structural violations sent to contextualization for the lint-based CWEs."},{"cited_title":"VC Formal: Formal Verification Solution | Synopsys,","cited_arxiv_id":null,"evidence_quote":"is the formal property verification tool that checks the custom assertion templates and produces falsified properties."}],"review_version":1}