{"id":"06e8281b-6222-41a9-bc0d-c89a8f015d11","arxiv_id":"2411.16111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMPirate uses eight LLMs to rewrite Verilog netlists into functionally equivalent circuits that evade GNN4IP, MOSS, Jplag, and SIM on most tested designs.","lead":"This paper presents LLMPirate, an automated method that uses large language models to rewrite hardware circuit designs into functionally identical versions that are not flagged by several common IP piracy detection tools. It shows that prompting LLMs with Boolean formulas and giving feedback can create enough structural change to fool current detectors, exposing a weakness in hardware IP protection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '100% evasion across every detection tool' claim is contradicted by the paper's own Table I (SIM 81.25%) and by Sec. V-J (GNN4IP flags GPS/MOR1KX), and the reported rates are best-over-LLM/best-over-strategy scores rather than a consistent configuration.","rationale":"I read the paper in good faith: LLMPirate is a plausible and technically interesting attack. The transformation dictionary approach, the divide-and-conquer handling of context limits, and the feedback-guided repair loop are well described, and the per-gate exhaustive equivalence check gives a reasonable functional-equivalence guarantee, with formal verification on sampled netlists as additional support. The problem is not the method itself but the claim that it 'consistently evade[s] detection on 100% of tested circuits across every detection tool.' That claim is contradicted by the paper's own Table I (SIM 81.25%), by Sec. V-F (25/32 for the best LLMs against SIM), and by Sec. V-J (GNN4IP flags GPS and MOR1KX). The figures aggregate the best similarity score over all LLMs and all mapping strategies, so the headline describes a best-case selection rather than a reproducible, configuration-independent result. The reader's weakest assumption about hand-chosen thresholds and non-native text modes is related and valid, but the more immediate internal inconsistency is that the reported aggregate itself does not support the '100%' wording. This does not warrant rejection: the underlying attack appears to work in a qualified, best-case sense, and the experimental data are sufficiently detailed to allow readers to see the per-LLM variation. A conditional acceptance requiring corrected claims, per-configuration results, and artifact release is the right outcome. I therefore keep the reader's CONDITIONAL verdict, with the caveat that the headline claim must be fixed before publication.","tokens_in":27850,"tokens_out":2840,"duration_ms":29506,"concrete_test":"Recompute the per-tool evasion counts under one fixed configuration, not a best-of-LLM/best-of-strategy aggregate: use a single LLM (e.g., GPT-4), the random mapping strategy, N=5 runs, the paper's stated thresholds (GNN4IP 0, MOSS 0.2, Jplag 0.3, SIM 0.3), and Jplag/SIM in text mode, over the 32 GNN4IP-repository netlists. Count how many netlists fall below threshold for each tool, and compare with Table I's 100%/100%/100%/81.25%. If any tool's count is below 32, the abstract's '100% across every detection tool' is false; if SIM remains at 26/32, the claim should be revised to state best-case evasion with SIM explicitly excepted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in the abstract and Sec. V-B is that LLMPirate 'consistently evade[s] detection on 100% of tested circuits across every detection tool' and 'successfully pirate[s] all 32 netlists against all four detection tools.' This is not supported by the paper's data under any single fixed configuration. Table I reports SIM evasion at 81.25%, i.e., 26/32 netlists, and the footnote attributes the shortfall to SIM's lack of native Verilog support. Sec. V-F states that even the best LLMs, GPT-3.5 and GPT-4, evade SIM for only 25 netlists. Sec. V-J reports that GNN4IP classifies all LLMPirate-generated versions of the GPS and MOR1KX netlists as pirated; the authors explain this away as GNN4IP's bias toward large netlists, but the abstract's phrasing 'across every detection tool' is still inaccurate for those large circuits. Moreover, the headline figures in Fig. 3 are explicitly the best (lowest) similarity scores over all mapping strategies and all eight LLMs; per-LLM results vary widely, e.g., CL-7B evades GNN4IP on only 10 of 32 netlists (Sec. V-C). Thus the central claim depends on an unstated best-case selection and on hand-chosen thresholds (GNN4IP 0, MOSS 0.2, Jplag 0.3, SIM 0.3) with Jplag and SIM run in text mode. The core rewriting method may well work, but the paper's headline overstates a typical or consistent success rate, and the claim as printed is internally inconsistent with the reported data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMPirate, an end-to-end LLM-based framework that rewrites gate-level Verilog netlists into functionally equivalent but structurally different versions, with the goal of evading hardware IP piracy detection. The method addresses three challenges: LLMs' poor handling of raw Verilog netlists (Solution A: syntax translation to Boolean functions), scalability to large netlists (Solution B: netlist characterization and divide-and-conquer prompting over gate types), and error-prone single-shot responses (Solution C: feedback-guided interactive refinement with up to M attempts). The authors evaluate eight LLMs on 31 netlists from the GNN4IP repository plus IBEX, MOR1KX, and a GPS module against four detection tools (GNN4IP, MOSS, Jplag, SIM). The headline claim is that LLMPirate consistently evades detection on 100% of tested circuits across every detection tool, with case studies on IBEX, MOR1KX, and GPS. The paper also includes an ablation study, overhead analysis, and an assessment of obfuscated netlists.","tokens_in":28199,"tokens_out":4239,"duration_ms":39634,"significance":"If the headline claims were fully supported, the paper would represent a significant contribution to hardware security: it would demonstrate that current IP piracy detectors, including a state-of-the-art GNN-based tool, are vulnerable to a black-box LLM-based rewriting attack that is automated, scalable, and does not require access to detector internals. The paper has several genuine strengths: the three proposed solutions are clearly motivated and the ablation study (Table V) shows that Solutions A and B are essential; functional equivalence is verified through exhaustive per-gate simulation and, for the GNN4IP repository netlists, through Cadence Conformal equivalence checking (Sec. VII-D); and the evaluation spans multiple detectors with different algorithmic principles. The significance is tempered, however, by the fact that the central '100% evasion' claim is contradicted by the paper's own data under any fixed configuration, and by the sensitivity of the results to hand-chosen thresholds and tool configurations.","major_comments":[{"comment":"The abstract states that LLMPirate 'consistently evade[s] detection on 100% of tested circuits across every detection tool' and Sec. V-B states that 'we are successfully able to pirate all 32 netlists against all four detection tools.' This is not supported by the paper's data. Table I reports only 81.25% evasion against SIM, with a footnote attributing the shortfall to SIM's lack of native Verilog support; Sec. V-F says that even the best LLMs, GPT-3.5 and GPT-4, evade SIM for only 25 of 32 netlists; and Sec. V-J reports that GNN4IP classifies all LLMPirate-generated versions of the GPS and MOR1KX netlists as pirated. The abstract's 'across every detection tool' wording is therefore inaccurate for SIM and for GNN4IP on the large case-study netlists, and the claim should be revised to state per-tool rates and to exclude or explain the large-netlist GNN4IP results.","section":"Abstract and Sec. V-B vs. Table I, Sec. V-F, Sec. V-J"},{"comment":"The headline performance in Fig. 3 is explicitly the best (lowest) similarity score over all eight LLMs and all five mapping strategies, as stated in footnote 5, and Sec. V-C further notes that per-LLM plots are for the best mapping strategy for each netlist and LLM. This best-case selection does not support the word 'consistent' in the abstract: under a fixed LLM and mapping strategy, per-LLM results vary widely, with Sec. V-C reporting that CL-7B evades GNN4IP on only 10 of 32 netlists and Llama3-8B on 11 of 32. The paper should report evasion rates under a fixed configuration (or a small set of configurations) and clearly characterize Fig. 3 as an upper bound, or change the claims to reflect that the reported success requires choosing the best LLM and mapping strategy per netlist.","section":"Sec. V-B, Fig. 3, and Sec. V-C"},{"comment":"The definition of 'evasion' depends on hand-chosen thresholds and tool configurations: GNN4IP threshold 0, MOSS 0.2, Jplag 0.3, SIM 0.3, with Jplag and SIM run in text mode because they do not natively support Verilog. These choices are load-bearing because the central contribution is measured against them; for example, the SIM shortfall in Table I is explained by text-mode operation. The paper should justify the threshold choices and provide a sensitivity analysis (e.g., varying thresholds over plausible ranges, and using native Verilog modes where available) to establish that the reported evasion rates are not artifacts of a single favorable configuration.","section":"Sec. V-A, Table III"}],"minor_comments":[{"comment":"The text says 'a dataset of 31 different Verilog netlists from the GNN4IP repository,' but Fig. 3 and the main text report results for 32 netlists; please clarify whether IBEX is the 32nd netlist and adjust the wording consistently.","section":"Sec. V-A"},{"comment":"The MOSS plots show single similarity scores because only one randomly picked pirated netlist per mapping strategy was queried, due to MOSS's query limit; the captions and figure axes should state explicitly that these are single samples, not distributions, to avoid visual over-interpretation.","section":"Sec. V-B and Fig. 5"},{"comment":"The claim that GNN4IP has a 'high bias towards classifying large netlists as pirated' is based on the observation that the similarity between the original GPS netlist and itself is 0.97; a more appropriate control would be the similarity between two different large netlists, and the current evidence does not directly demonstrate a size-bias in GNN4IP.","section":"Sec. V-J"},{"comment":"The sentence 'we check the equivalence of a successfully pirated netlist against each of the four detection tools' appears to be a wording error, since equivalence should be checked against the original netlist rather than against detection tools.","section":"Sec. VII-D"},{"comment":"The footnote and Sec. V-F describe SIM's high scores as a 'false-positive rate'; this is technically a high-similarity-score issue caused by text-mode tokenization, not a false-positive rate, and the terminology should be corrected.","section":"Table I footnote, Sec. V-F"},{"comment":"Minor presentation issues: inconsistent capitalization of 'Jplag'/'JPlag' and 'Verilog' vs 'verilog', and Sec. V-H cites a '700% improvement' for Llama3-8B without referencing the exact figure axis from which this value is read.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's core attack idea appears sound and the experimental apparatus is substantial, but the abstract and Sec. V-B overstate the results in a way that is internally inconsistent with the paper's own tables. In my view the appropriate path is a major revision that: (1) rewrites the headline claims to report per-tool success rates under a fixed configuration, (2) recharacterizes Fig. 3 as best-case performance, and (3) adds threshold sensitivity analysis. These are fixable within the manuscript's scope, so I do not recommend rejection, but the current version should not be published as-is because a reader relying on the abstract would be misled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the core attack is real: LLMPirate rewrites gate-level netlists into functionally equivalent variants that dodge GNN4IP, MOSS, and Jplag on most small benchmarks and SIM on most circuits. The authors do the right thing by exhaustively checking every gate replacement and by running Cadence Conformal on small netlists; that is solid evidence the transformations preserve functionality. Second, the headline claim of 100% evasion across every detection tool is false as stated. The paper's own Table I gives SIM 81.25%, and Sec. V-J admits GNN4IP flags the GPS and MOR1KX netlists as pirated. The abstract should say best-case on most circuits, not consistently evade detection on 100%.\n\nWhat is new here is the combination rather than any single component: Boolean-function prompt translation, divide-and-conquer over gate types, and feedback-guided retries, applied against four structurally different detectors. Prior work used reinforcement learning or genetic programming against one detector; using off-the-shelf LLMs this way is genuinely new. The ablation study is useful, showing Solutions A and B are essential and that the interactive feedback (Solution C) improves SIM evasion from 7 to 26 netlists.\n\nThe main soft spot is the gap between the abstract and the data. Figure 3 and Sec. V-B use best-over-LLM and best-over-strategy scores (footnote 5), so the 100% figure is a post-hoc selection. Per-LLM numbers are far weaker: CL-7B evades GNN4IP only 10 of 32 netlists, Llama3-8B 11 of 32. The thresholds for all four tools are chosen by the authors, and Jplag and SIM are run in text mode rather than with native Verilog support, so success rates are configuration-dependent. That is not a fatal flaw, any attack evaluation needs thresholds, but it should be reported transparently, not as a blanket 100%.\n\nOne more minor point: I do not see any mention of releasing code or artifacts. Given the breadth of the evaluation, an artifact release would materially help reproducibility and trust. The case studies on IBEX, MOR1KX, and GPS are good, but the text calls them successful pirates even though GNN4IP fails on the two large netlists; that mismatch should be reconciled.\n\nWho this is for: hardware security researchers, especially people working on GNN4IP or IP piracy detection, and anyone interested in LLM-based code rewriting as an attack primitive. It deserves a serious referee. My recommendation: send it to peer review with major comments requiring an accurate abstract, honest per-configuration results, and ideally an artifact release.","headline":"The core LLM-based netlist-rewriting attack is real and worth engaging, but the paper's '100% evasion' headline is contradicted by its own tables and needs correction before publication.","tokens_in":747,"tokens_out":1031,"would_cite":true,"duration_ms":32235,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMPirate, the first LLM-based technique for hardware IP piracy, rewrites gate-level Verilog netlists into functionally equivalent forms that evade four state-of-the-art piracy detection tools.","keywords":["hardware IP piracy","large language models","Verilog netlist","circuit rewriting","piracy detection evasion","GNN4IP","similarity detection","functional equivalence"],"falsifier":"A decisive check is to rerun LLMPirate's pirated netlists through Jplag and SIM with Verilog-aware modes enabled or with thresholds below 0.3, and to formally verify the pirated IBEX, MOR1KX, and GPS netlists; if any large netlist is flagged or fails equivalence, the blanket evasion and equivalence claims do not hold.","tokens_in":27608,"feed_emoji":"🔓","tokens_out":9934,"duration_ms":86156,"temperature":0.7,"pith_summary":"LLMPirate is an end-to-end automated attack that uses a large language model to rewrite a gate-level Verilog netlist into a functionally equivalent version that current IP piracy detectors do not flag. If the authors are right, anyone with black-box access to an LLM and a detector can pirate valuable chip designs without touching the tools' internals. The method works by translating gates into Boolean formulas, rewriting each gate type separately to fit LLM context limits, and using feedback to fix errors. The paper reports per-tool evasion of 100% for GNN4IP, MOSS, and Jplag and 81.25% for SIM on the tested netlists, and it demonstrates the flow on IBEX, MOR1KX, and a GPS module, while also noting that GNN4IP flags those two largest netlists and scores the originals 0.97 against themselves.","feed_headline":"LLMPirate rewrites circuits to slip past piracy detectors","feed_subtitle":"Reported LLM-based netlist rewriting beats GNN4IP, MOSS, Jplag, and SIM on most tested designs, including processors.","key_machinery":"The central mechanism is a per-gate-type transformation dictionary: each unique gate type in the netlist is first translated from Verilog syntax into a generic Boolean-function line such as 'c = AND(a,b)', the LLM is prompted to rewrite that line using only an allowed complete set of operators, and the response is validated by syntax, operator, and exhaustive functional checks with interactive feedback. The dictionary of valid transformations is then applied gate-by-gate through five mapping strategies, named AND-NOT, NAND, NOR, OR-NOT, and random, which is what actually produces a structurally different but functionally equivalent pirated netlist.","core_discovery":"The paper's central claim is that hardware IP piracy can be carried out as a black-box LLM prompting task. Given an original gate-level netlist, LLMPirate characterizes the set of gate types, translates one representative of each type into a Boolean-function line, and asks the LLM to rewrite it using only an allowed list of Boolean operators, for example replacing an OR gate with NAND or NOR-of-inverted-input forms. Each candidate rewrite is checked for syntax, operator compliance, and functional equivalence by exhaustive simulation, and the LLM is allowed up to five feedback-guided attempts, so the final dictionary of transformations is guaranteed to preserve behavior. Applying these transformations gate-by-gate with five mapping strategies yields pirated netlists that stay below the detection thresholds of GNN4IP, MOSS, Jplag, and SIM on most or all of the 32 test designs; the authors also pirate the IBEX and MOR1KX processors and a GPS module, with functional equivalence of the small benchmarks confirmed by a commercial formal equivalence checker.","pith_inferences":["The paper's own data qualify the '100% evasion' headline: Table I reports SIM evasion at 81.25%, and Section V-J reports that GNN4IP labels the GPS and MOR1KX netlists as pirated, so a reader should read the headline as the paper's stated goal rather than an unqualified measured result.","The evasion numbers are threshold-dependent and configuration-dependent; if defenders run Jplag and SIM in Verilog-aware modes or tune thresholds lower, the measured success rates would likely drop, so the method's practical impact should be assessed across threshold settings.","The same divide-and-conquer prompting with feedback could likely be extended to other hardware description styles, such as VHDL or register-transfer-level code, and to higher-level transformations such as factoring or retiming, which would test whether the attack generalizes beyond simple gate substitution.","Because GNN4IP scores the original GPS and MOR1KX netlists 0.97 against themselves, detector comparisons on large designs need size-calibrated baselines; otherwise detector bias, not attack strength, determines the reported evasion."],"forward_implications":["A would-be pirate with no access to detector internals can automate the entire rewrite-and-evaluate loop, so source-similarity-based IP protection is not a sufficient defense against LLM-equipped adversaries.","Real designs such as processors and a GPS module can be pirated in minutes, so the threat is practical for large netlists, not just small benchmarks.","Because the transformation dictionary is built once per gate type and reused, the marginal cost of pirating an additional netlist is small, making large-scale theft feasible.","The feedback-guided interaction makes even small open-source LLMs useful for the attack, so the barrier to entry is not limited to the largest proprietary models."],"supporting_citations":[{"why":"Defines GNN4IP, the machine-learning detector that LLMPirate must evade.","marker":"[78]"},{"why":"Supplies MOSS, the winnowing-based similarity detector with the 0.2 threshold.","marker":"[2]"},{"why":"Supplies Jplag, the greedy-string-tiling detector run in text mode.","marker":"[38]"},{"why":"Supplies SIM, the token-overlap detector run in text mode.","marker":"[32]"},{"why":"Provides the 31 benchmark netlists used for the main evaluation.","marker":"[1]"},{"why":"Provides VeriGen, the Verilog-fine-tuned LLM whose failure motivates LLMPirate's syntax translation approach.","marker":"[74]"},{"why":"Provides AttackGNN, the closest prior black-box evasion technique that LLMPirate compares itself against.","marker":"[27]"}],"fun_headline_variants":["LLMPirate: LLMs rewrite chips to evade IP piracy checks","LLM-based tool rewrites netlists, fooling piracy detectors","LLMPirate evades 100% of hardware piracy detectors","LLMs rewrite circuits to pirate hardware, evading detectors","LLMPirate: black-box hardware piracy via LLM rewriting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the chosen detection thresholds and text-mode configurations represent how practitioners actually run these tools; if defenders use different thresholds or Verilog-aware modes, the measured evasion rates could change.","fun_headline_variants_meta":{"raw":{"variants":["LLMPirate: LLMs rewrite chips to evade IP piracy checks","LLM-based tool rewrites netlists, fooling piracy detectors","LLMPirate evades 100% of hardware piracy detectors","LLMs rewrite circuits to pirate hardware, evading detectors","LLMPirate: black-box hardware piracy via LLM rewriting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2365,"prompt_tokens":1068,"completion_tokens":1297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":1207}},"tokens_in":684,"tokens_out":1297,"duration_ms":9074,"temperature":1.0,"reasoning_tokens":1207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:32:35.000531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to rerun LLMPirate's pirated netlists through Jplag and SIM with Verilog-aware modes enabled or with thresholds below 0.3, and to formally verify the pirated IBEX, MOR1KX, and GPS netlists; if any large netlist is flagged or fails equivalence, the blanket evasion and equivalence claims do not hold.","supporting_citations":[{"cited_title":"GNN4IP: Graph Neural Network for Hardware Intellectual Property Piracy Detec- tion,","cited_arxiv_id":null,"evidence_quote":"Defines GNN4IP, the machine-learning detector that LLMPirate must evade."},{"cited_title":"A System for Detecting Software Similarity,","cited_arxiv_id":null,"evidence_quote":"Supplies MOSS, the winnowing-based similarity detector with the 0.2 threshold."},{"cited_title":"JPlag - Detecting Software Plagiarism,","cited_arxiv_id":null,"evidence_quote":"Supplies Jplag, the greedy-string-tiling detector run in text mode."},{"cited_title":"The software and text similarity tester SIM,","cited_arxiv_id":null,"evidence_quote":"Supplies SIM, the token-overlap detector run in text mode."},{"cited_title":"HW2VEC: A Graph Learning Tool for Automating Hard- ware Security,","cited_arxiv_id":null,"evidence_quote":"Provides the 31 benchmark netlists used for the main evaluation."},{"cited_title":"Verigen: A large language model for verilog code generation,","cited_arxiv_id":null,"evidence_quote":"Provides VeriGen, the Verilog-fine-tuned LLM whose failure motivates LLMPirate's syntax translation approach."},{"cited_title":"AttackGNN: Red-Teaming GNNs in Hardware Security Using Reinforcement Learning","cited_arxiv_id":"2402.13946","evidence_quote":"Provides AttackGNN, the closest prior black-box evasion technique that LLMPirate compares itself against."}],"review_version":1}