REVIEW 5 major objections 5 minor 28 references
Wrong Code, Right Structure: Learning Netlist Representations from Imperfect LLM-Generated RTL
T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Even functionally incorrect LLM-generated chip code can train netlist models that match or beat clean labeled data.
desk verdict The core idea is novel but the paper overstates it: the 'wrong code, right structure' premise is never directly verified because functional correctness is never measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the netlist-level structural similarity filter. Each generated netlist and its golden reference are reduced to graph-level embeddings by summing mean and max pooling over per-gate initial features (gate type, connectivity to primary inputs/outputs, in-degree/out-degree); cosine similarity between the two embeddings decides retention. A single unreported threshold tau separates useful noisy designs from irrelevant ones. A complementary LLM-based architecture voting step selects diverse implementations that would fail the filter, so the corpus mixes fidelity to the reference structure with architectural breadth. The GNN then learns from this corpus through sampled mess
What would settle it
Take a single arithmetic function, generate a large batch of LLM variants, sort them by the paper's cosine similarity to a golden netlist, and simulate each to measure functional correctness. If the correlation between similarity and functional correctness is near zero—or if a model trained on the bottom half of the similarity ranking outperforms one trained on the top half on a held-out architectural variant—the structural-similarity filter is not the explanation for the reported gains.
Extended reading notes
Core claim
The paper's central claim is that structural and functional information in a netlist are partially decoupled: functionally wrong LLM-generated RTL can have wrong truth tables yet still synthesize to netlists whose graph topology preserves the characteristic patterns of the intended operation. Using a two-part selection process—a similarity filter that keeps netlists close to a golden reference and an LLM-voting step that admits architecturally distinct implementations—the authors build a training corpus from imperfect code. A graph neural network trained on this corpus matches or exceeds a model trained on scarce high-quality labels on operator-level sub-circuit identification, and, on an IP
Load-bearing premise
The claim collapses if cosine similarity between mean-and-max-pooled gate features of a generated netlist and a golden netlist is not a faithful proxy for functionally relevant structure, because the threshold tau (whose value is not reported) is what separates useful noisy designs from irrelevant ones and could instead be selecting for similarity to the reference architecture's statistics.
Editorial extensions
If this is right
- Models trained on LLM-generated noisy netlists generalize to real-world netlists, matching or surpassing those trained on scarce high-quality annotated data.
- Sub-circuit boundary identification can be extended from operator-level blocks to IP-level modules such as CPU cores in SoCs, a scale previously impractical due to labeling cost.
- The structural-similarity filter is responsible for the precision gain; without it, performance drops (in the IP case, F1 falls from 68.35% to 60.44%).
- LLM-voted architectural diversity improves generalization to unseen architectures of the same function, outperforming both the baseline and unfiltered LLM data.
Reading between the lines
- If structural patterns are indeed robust to functional errors, the same data-augmentation recipe could apply to other hardware-security tasks such as trojan detection or IP piracy, where annotated netlists are equally scarce.
- The paper's filtering uses the golden netlist as an anchor; a natural extension is to use an ensemble of reference implementations or a learned similarity metric that no longer requires a golden design, allowing fully spec-driven generation without any reference RTL.
- The threshold tau is a single empirical hyperparameter; one could test whether a per-class or per-complexity adaptive threshold yields a better diversity-fidelity trade-off.
- A two-stage curriculum—first learn from structurally similar noisy netlists, then from architecturally distinct ones—might further improve representation robustness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an end-to-end framework that uses LLMs to generate RTL from functional specifications, synthesizes the RTL into netlists, and trains a GNN (based on GNN-RE) on these netlists for sub-circuit boundary identification and component classification. The central claim is that even when LLM-generated RTL is functionally imperfect, the synthesized netlists retain structural patterns indicative of the intended functionality, so that a noisy synthetic corpus can substitute for scarce, expensive real-world labeled netlists. The authors report operator-level experiments on arithmetic modules, an ablation of an LLM-based architecture-voting mechanism, and an IP-level case study in which a model trained on augmented PicoRV32 data is tested on NEORV32.
Significance. If established, the claim would be practically valuable: it would offer a low-cost way to break the data bottleneck in netlist representation learning and would extend sub-circuit identification from operator-level to IP-level scale. The paper also has real strengths: it describes a concrete, automatable pipeline; it deliberately reuses the GNN-RE backbone to isolate the contribution of data augmentation; and it includes an IP-level generalization experiment with a strict train/test domain separation. However, the core empirical premise — that the generated RTL is actually functionally imperfect — is never directly verified, and the reported experimental evidence is not statistically grounded. These issues are load-bearing for the paper's central contribution.
major comments (5)
- [Sec. 1 and Sec. 3.2.1] The paper's central claim is that functionally imperfect LLM-generated RTL still yields structurally informative netlists. Yet the manuscript never measures functional correctness of any generated corpus. Sec. 1 explicitly states that the framework bypasses strict functional verification, and Sec. 3.2.1 describes only a synthesis-feedback loop that ensures synthesizability, not functional equivalence. No simulation, assertion-based verification, or formal-equivalence results are reported. Consequently, the 'noisy synthetic corpus' could consist mostly of correct RTL, in which case the results reduce to 'LLMs can generate useful training RTL' and the 'wrong code, right structure' observation is untested. This is a load-bearing omission and should be fixed by reporting functional error rates (e.g., via simulation against golden designs) and, ideally, ablating on subsets stratified by funct
- [Sec. 3.2.2, Eq. (2)-(3)] The structural-similarity filter depends on a threshold tau that is never reported or analyzed. The filter assumes that cosine similarity between mean+max pooled initial gate features is a faithful proxy for functional relevance, and that one empirically chosen threshold can separate useful noisy designs from irrelevant ones. Without reporting tau and without a sensitivity study across a range of thresholds, the reader cannot assess whether the reported gains are robust or an artifact of a particular filtering point. At minimum, the value of tau used in Tables 1-3 and an ablation over tau should be provided.
- [Table 1] The operator-level evaluation cherry-picks the best of five generated datasets: the abstract and Sec. 1 claim that models trained on noisy synthetic data 'match or even surpass' methods trained on high-quality data, but of the five runs, LLM-Aug-t4 is substantially worse than the baseline (F1-Macro 84.62 vs. 90.15), and no error bars or multiple-seed variation are reported anywhere in the paper. The claim therefore rests on one favorable draw. The authors should report the mean and standard deviation over all five generated sets (or over multiple seeds with a fixed dataset) and perform a statistical significance test when comparing to the baseline.
- [Sec. 3.2.3] The architecture-voting mechanism uses the same LLM that generated the RTL to evaluate its own output, asking for scores based on 'architectural diversity' and 'implementation complexity.' This is a self-referential selection procedure: it may be enriching for outputs that the LLM judges as well-structured, potentially biasing toward functionally correct or at least LLM-preferred designs. The paper does not compare voting against random selection or against a non-LLM baseline, so its benefit is confounded with the LLM's self-evaluation biases. Please add a control experiment comparing voting to random selection of the same number of designs, or report the distribution of functional correctness in voted vs. non-voted sets.
- [Table 3] The IP-level experiment, while a valuable stress test, has confounds that prevent a clean attribution of the result to the proposed filtering mechanism. LLM-Filtered and LLM-Raw differ both in the filtering operation and in the number of training graphs (17,427 vs. 19,302 nodes); Rule-Based is a different augmentation family without a matched training scale. Moreover, only one target SoC is used, so the generality of the 68.35% F1 result is unknown. To support the claim that the structural-similarity filter causes the improvement, the comparison should match training-set scale and report results over several target SoCs or at least several train/test splits.
minor comments (5)
- [Sec. 4.2] There is an incomplete sentence: 'Note that in this section, the input of our framework is only the functional .' Presumably 'functional specification' is intended.
- [Sec. 3.2.3] The symbol N is used both for the number of candidate designs in architecture voting (N=10) and for the graph node set elsewhere; please disambiguate.
- [Sec. 4.1.1 / Sec. 4.4] Several hyperparameters relevant to reproducibility are missing or only partially specified: the exact prompt templates for generation and voting, the number of generation attempts per specification, the debug-agent repair loop iterations, and the precise synthesis tool options. These should be documented in an appendix or supplementary material.
- [Sec. 3.3.1] The graph is described as undirected, while GNN-RE and several related netlist models treat gate-level netlists as directed graphs. The choice is not justified and may affect the kinds of structural patterns the GNN can capture; please clarify whether directionality is encoded in the node features instead.
- [References] Reference [27] contains the placeholder '<git-hash>' rather than a concrete commit identifier. The NEORV32 reference also lacks a version/commit and access date is inconsistent with the manuscript's 2026 date.
Circularity Check
No significant circularity: central generalization results are evaluated on unseen data and do not reduce to fitted inputs by construction.
full rationale
The paper's central derivation is not circular. The claimed value of imperfect LLM-generated RTL is tested empirically: models are trained on augmented netlists and evaluated on operator-level test sets from GNN-RE and, more importantly, on the unseen NEORV32 SoC, which was never used during generation, filtering, or training. The structural-similarity filter (Sec. 3.2.2, Eq. 2-3) selects training examples by cosine similarity to golden netlists using initial node features; this is an input-selection heuristic, not a construction that forces the downstream classification outputs to equal those similarities. The architecture-voting mechanism (Sec. 3.2.3) uses an LLM to evaluate LLM-generated designs, but this self-referential selection is validated by an ablation (Table 2) and does not mathematically entail the reported generalization gains. The paper contains self-citations (e.g., [23] for LLM evaluator quality), but the claim is also supported by external references [24,25] and by the paper's own experiments. A methodological concern is that functional correctness of generated RTL is never measured, so the 'wrong code' premise is not directly verified; however, this is a support/correctness risk, not circularity. The unreported threshold tau and the use of golden netlists for filtering may raise data-leakage concerns, but they do not reduce any prediction to its inputs by definition or by fitted-parameter renaming.
Assumptions & free parameters
free parameters (5)
- tau (structural similarity threshold) =
not reported (set empirically)
- architecture voting N and k =
N=10, k=3
- curation split ratio (structural-filter vs voting) =
50/50
- cell-count outlier filter threshold =
not specified
- LLM sampling temperature =
0.8
assumptions (4)
- ad hoc to paper Functionally imperfect LLM-generated RTL, after synthesis, preserves structural patterns indicative of the intended functionality.
- domain assumption Cosine similarity of mean+max pooled initial gate features (Eq. 2-3) is a valid proxy for functional relevance.
- domain assumption LLMs are reliable evaluators of RTL design quality when voting on architectural diversity and implementation complexity.
- domain assumption Module boundaries in generated hierarchical designs provide correct functional labels even when the design contains functional errors.
Cite this review
Pith. "Pith review of Wrong Code, Right Structure: Learning Netlist Representations from Imperfect LLM-Generated RTL." pith.science (2026). https://pith.science/paper/56WZJNVC
@misc{pith2026260309161,
author = {Pith},
title = {Pith review of: Wrong Code, Right Structure: Learning Netlist Representations from Imperfect LLM-Generated RTL},
year = {2026},
howpublished = {\url{https://pith.science/paper/56WZJNVC}},
note = {Machine review of arXiv:2603.09161}
}
read the original abstract
Learning effective netlist representations is fundamentally constrained by the scarcity of labeled datasets, as real designs are protected by Intellectual Property (IP) and costly to annotate. Existing work therefore focuses on small-scale circuits with clean labels, limiting scalability to realistic designs. Meanwhile, Large Language Models (LLMs) can generate Register-Transfer-Level (RTL) at scale, but their functional incorrectness has hindered their use in circuit analysis. In this work, we make a key observation: even when LLM-Generated RTL is functionally imperfect, the synthesized netlists still preserve structural patterns that are strongly indicative of the intended functionality. Building on this insight, we propose a cost-effective data augmentation and training framework that systematically exploits imperfect LLM-Generated RTL as training data for netlist representation learning, forming an end-to-end pipeline from automated code generation to downstream tasks. We conduct evaluations on circuit functional understanding tasks, including sub-circuit boundary identification and component classification, across benchmarks of increasing scales, extending the task scope from operator-level to IP-level. The evaluations demonstrate that models trained on our noisy synthetic corpus generalize well to real-world netlists, matching or even surpassing methods trained on scarce high-quality data and effectively breaking the data bottleneck in circuit representation learning.
Figures
Reference graph
Works this paper leans on
-
[1]
Gnn4tj: Graph neural networks for hardware trojan detection at register transfer level
Rozhin Yasaei, Shih-Yuan Yu, and Mohammad Abdullah Al Faruque. Gnn4tj: Graph neural networks for hardware trojan detection at register transfer level. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1504–1509. IEEE, 2021. Wrong Code, Right Structure: Learning Netlist Representations from Imperfect LLM-Generated RTL Confer...
2021
-
[2]
Graph similarity and its applications to hardware security.IEEE Transactions on Computers, 69(4):505–519, 2019
Marc Fyrbiak, Sebastian Wallat, Sascha Reinhard, Nicolai Bissantz, and Christof Paar. Graph similarity and its applications to hardware security.IEEE Transactions on Computers, 69(4):505–519, 2019
2019
-
[3]
Wenji Fang, Jing Wang, Yao Lu, Shang Liu, and Zhiyao Xie. Geneda: Unleash- ing generative reasoning on netlist via multimodal encoder-decoder aligned foundation model.arXiv preprint arXiv:2504.09485, 2025
arXiv 2025
-
[4]
Widegate: Beyond directed acyclic graph learning in subcircuit boundary prediction
Jiawei Liu, Zhiyan Liu, Xun He, Jianwang Zhai, Zhengyuan Shi, Qiang Xu, Bei Yu, and Chuan Shi. Widegate: Beyond directed acyclic graph learning in subcircuit boundary prediction. In2025 Design, Automation & Test in Europe Conference (DATE), pages 1–7, 2025
2025
-
[5]
Relut-gnn: Reverse engineering data path elements from lut netlists using graph neural networks
Kishore Pula, Aparajithan Nathamuni Venkatesan, Ram Venkat Narayanan, Sun- darakumar Muthukumaran, Ranga Vemuri, and John Emmert. Relut-gnn: Reverse engineering data path elements from lut netlists using graph neural networks. In 2023 IEEE 66th International Midwest Symposium on Circuits and Systems (MWS- CAS), pages 511–515. IEEE, 2023
2023
-
[6]
Ziyi Wang, Chen Bai, Zhuolun He, Guangliang Zhang, Qiang Xu, Tsung-Yi Ho, Yu Huang, and Bei Yu. Fgnn2: A powerful pretraining framework for learning the logic functionality of circuits.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 44(1):227–240, 2025
2025
-
[7]
Functionality matters in netlist representation learning
Ziyi Wang, Chen Bai, Zhuolun He, Guangliang Zhang, Qiang Xu, Tsung-Yi Ho, Bei Yu, and Yu Huang. Functionality matters in netlist representation learning. InProceedings of the 59th ACM/IEEE Design Automation Conference, pages 61–66, 2022
2022
-
[8]
Gnn-re: Graph neural networks for reverse engineering of gate-level netlists.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 41(8):2435–2448, 2022
Lilas Alrahis, Abhrajit Sengupta, Johann Knechtel, Satwik Patnaik, Hani Saleh, Baker Mohammad, Mahmoud Al-Qutayri, and Ozgur Sinanoglu. Gnn-re: Graph neural networks for reverse engineering of gate-level netlists.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 41(8):2435–2448, 2022
2022
Show all 28 references
-
[9]
Dagnn-re: Directed acyclic graph neural network for functional reverse engineering of gate-level netlist.Integration, 102:102343, 2025
Zongtai Li, Liang Yang, and Mian Lou. Dagnn-re: Directed acyclic graph neural network for functional reverse engineering of gate-level netlist.Integration, 102:102343, 2025
2025
-
[10]
Appgnn: Approximation-aware functional reverse en- gineering using graph neural networks
Tim Bücher, Lilas Alrahis, Guilherme Paim, Sergio Bampi, Ozgur Sinanoglu, and Hussam Amrouch. Appgnn: Approximation-aware functional reverse en- gineering using graph neural networks. InProceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, pages 1–9, 2022
2022
-
[11]
Trojan-guard: Hardware trojans detection using gnn in rtl designs.arXiv preprint arXiv:2506.17894, 2025
Kiran Thorat, Amit Hasan, Caiwen Ding, and Zhijie Shi. Trojan-guard: Hardware trojans detection using gnn in rtl designs.arXiv preprint arXiv:2506.17894, 2025
2025 arXiv
-
[12]
Hw2vec: A graph learning tool for automating hard- ware security
Shih-Yuan Yu, Rozhin Yasaei, Qingrong Zhou, Tommy Nguyen, and Moham- mad Abdullah Al Faruque. Hw2vec: A graph learning tool for automating hard- ware security. In2021 IEEE International Symposium on Hardware Oriented Security and Trust (HOST), pages 13–23. IEEE, 2021
2021
-
[13]
Hardware trojan detection using graph neural networks.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 44(1):25–38, 2022
Rozhin Yasaei, Luke Chen, Shih-Yuan Yu, and Mohammad Abdullah Al Faruque. Hardware trojan detection using graph neural networks.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 44(1):25–38, 2022
2022
-
[14]
Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution
Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution. In2024 IEEE LLM Aided Design Workshop (LAD), page 1–5. IEEE, June 2024
2024
-
[15]
Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework
Kaiyan Chang, Kun Wang, Nan Yang, Ying Wang, Dantong Jin, Wenlong Zhu, Zhirong Chen, Cangyuan Li, Hao Yan, Yunhao Zhou, et al. Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework. InProceedings of the 61st ACM/IEEE Design A...
2024
-
[16]
Rtllm: An open-source bench- mark for design rtl generation with large language model, 2023
Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. Rtllm: An open-source bench- mark for design rtl generation with large language model, 2023
2023
-
[17]
Gnn4ip: Graph neural network for hardware intellectual property piracy detection
Rozhin Yasaei, Shih-Yuan Yu, Emad Kasaeyan Naeini, and Mohammad Abdullah Al Faruque. Gnn4ip: Graph neural network for hardware intellectual property piracy detection. In2021 58th ACM/IEEE Design Automation Conference (DAC), pages 217–222. IEEE, 2021
2021
-
[18]
Hansen, H
M. Hansen, H. Yalcin, and J. Hayes. ISCAS High-Level Models. http://web.eecs. umich.edu/~jhayes/iscas.restore/benchmark.html. Accessed: Sep. 13, 2021
2021
-
[19]
The epfl combinational benchmark suite.Hypotenuse, 256(128):214335, 2015
Luca Amarú, Pierre-Emmanuel Gaillardon, and Giovanni De Micheli. The epfl combinational benchmark suite.Hypotenuse, 256(128):214335, 2015
2015
-
[20]
Deepgate2: Functionality- aware circuit representation learning
Zhengyuan Shi, Hongyang Pan, Sadaf Khan, Min Li, Yi Liu, Junhua Huang, Hui- Ling Zhen, Mingxuan Yuan, Zhufei Chu, and Qiang Xu. Deepgate2: Functionality- aware circuit representation learning. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pages 1–9...
2023
-
[21]
Deepgate3: Towards scalable circuit representation learning
Zhengyuan Shi, Ziyang Zheng, Sadaf Khan, Jianyuan Zhong, Min Li, and Qiang Xu. Deepgate3: Towards scalable circuit representation learning. InProceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pages 1–9, 2024
2024
-
[22]
Functional matching of logic subgraphs: Beyond structural isomorphism.arXiv preprint arXiv:2505.21988, 2025
Ziyang Zheng, Kezhi Li, Zhengyuan Shi, and Qiang Xu. Functional matching of logic subgraphs: Beyond structural isomorphism.arXiv preprint arXiv:2505.21988, 2025
2025
-
[23]
Autosilicon: Scaling up rtl design generation capability of large language models.ACM Transactions on Design Automation of Electronic Systems, 30(6):1–21, 2025
Cangyuan Li, Chujie Chen, Yudong Pan, Wenjun Xu, Yiqi Liu, Kaiyan Chang, Yujie Wang, Mengdi Wang, Huawei Li, Yinhe Han, et al. Autosilicon: Scaling up rtl design generation capability of large language models.ACM Transactions on Design Automation of Electronic Systems, 30(6):1...
2025
-
[24]
Llm voting: Human choices and ai collective decision-making
Joshua C Yang, Damian Dailisan, Marcin Korecki, Carina I Hausladen, and Dirk Helbing. Llm voting: Human choices and ai collective decision-making. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 1696–1708, 2024
2024
-
[25]
Embodied llm agents learn to cooperate in organized teams.arXiv preprint arXiv:2403.12482, 2024
Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L Griffiths, and Mengdi Wang. Embodied llm agents learn to cooperate in organized teams.arXiv preprint arXiv:2403.12482, 2024
2024 arXiv
-
[26]
Graphsaint: Graph sampling based inductive learning method.arXiv preprint arXiv:1907.04931, 2019
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method.arXiv preprint arXiv:1907.04931, 2019
1907 arXiv
-
[27]
PicoRV32 - A Size-Optimized RISC-V CPU Core
Clifford Wolf. PicoRV32 - A Size-Optimized RISC-V CPU Core. https://github. com/YosysHQ/picorv32, 2015. Commit <git-hash>
2015
-
[28]
NEORV32: A small, customizable and extensible mcu-class 32-bit risc-v soft-core cpu and microcontroller-like soc written in platform-independent vhdl
stnolting. NEORV32: A small, customizable and extensible mcu-class 32-bit risc-v soft-core cpu and microcontroller-like soc written in platform-independent vhdl. https://github.com/stnolting/neorv32, 2025. Accessed: 2025-11-19
2025
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.