REVIEW 3 major objections 7 minor 3 cited by
ForgeEDA: A Comprehensive Multimodal Dataset for Advancing EDA
T0 review · 3 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A new open dataset of 1,189 chip designs provides 83,155 logic graphs and shows more training data improves AI for EDA.
desk verdict A useful multimodal EDA dataset whose AI scaling claims are undercut by the lack of a documented held-out evaluation split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the dataset itself, with its four linked circuit representations: RTL source code, post-mapping netlists, placed netlists, and And-Inverter Graphs, along with 83,155 randomly extracted sub-AIGs of 500-5,000 nodes stored as graph representations for PyTorch Geometric. The mechanism that makes it useful is the explicit alignment of all formats and their accompanying synthesis, placement, and timing reports to a single design, so the same circuit can be studied at each stage of the EDA flow. This alignment supports both benchmarking of EDA tools on identical input/output pairs and supervised training of circuit-learning models on graph structure plus PPA labels.
What would settle it
A concrete check is to retrain a circuit model on a matched set of full-chip synthesized netlists: if the data-scaling gains seen on ForgeEDA's module-level sub-AIGs disappear, the module-as-top-level shortcut is not a valid proxy for real design diversity.
Extended reading notes
Core claim
ForgeEDA is a multimodal circuit dataset that binds four representations of each design into one resource: RTL source code, post-mapping netlists, placed netlists with physical-design reports, and And-Inverter Graphs. The construction expands coverage by collecting repositories via datasheet-derived keywords and by synthesizing each module as a separate top module, turning 1,189 repositories into 4,450 netlists. The paper evaluates the dataset in two ways: it benchmarks logic synthesis and AIG optimization, exposing a substantial performance gap between open-source and commercial tools, and it trains representative circuit-learning models on the graphs, showing that model error decreases monotonically as the training subset grows from 1% to 10% to the full dataset. The conclusion is that the scale and diversity of training data are decisive for AI4EDA and that ForgeEDA supplies the missing resource.
Load-bearing premise
The dataset's comprehensiveness rests on the assumption that the 1,189 synthesizable Verilog repositories collected from the internet via datasheet keywords are representative of real-world chip design diversity, while treating each module as a separate top module assumes module-level synthesis statistics are meaningful proxies for full-chip PPA.
Editorial extensions
If this is right
- Researchers can use ForgeEDA's 4,450 netlists and 83,155 sub-AIGs to train and evaluate circuit representation models, with evidence that larger training subsets lower prediction error.
- Logic-synthesis benchmarking is no longer confined to near-saturated small benchmarks; the dataset reveals that open-source synthesis tools lag commercial tools by roughly 1.77x in area and 1.20x in delay on average.
- The multimodal alignment lets users study how design decisions propagate across synthesis and physical design stages, including timing reports extracted after placement.
- Public release gives researchers a common ground for comparing PPA prediction and logic-optimization methods, potentially exposing where academic tools fail on real-world designs.
Reading between the lines
- Because each module is synthesized separately as a top module, the netlists are mostly block-level rather than full-system; gains from training on sub-AIGs may not transfer to whole-chip netlists, and a full-chip companion set would be a natural extension.
- The monotone data-scaling curves invite a formal study of scaling laws for circuit representation learning, potentially predicting how much data a target error requires.
- All PPA reports come from one 130nm process design kit, so models trained on them may need fine-tuning to transfer to other process nodes or FPGA mappings.
- The paired RTL code and PPA labels could support fine-tuning large language models to generate synthesizable RTL that is aware of downstream area and delay, an application the paper does not explore.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ForgeEDA is a proposed open-source multimodal circuit dataset. It collects 1,189 Verilog repositories from the internet based on datasheet-derived keywords, filters them for synthesizability, and synthesizes each module as a top module with Synopsys Design Compiler and the SkyWater 130nm PDK, producing 4,450 post-mapping netlists, 4,450 placed netlists, 4,450 AIGs, and 83,155 sub-AIGs with PPA and timing reports. The paper benchmarks DCU against Yosys on RTL synthesis, compares several ABC synthesis and mapping scripts, and trains five GNN models on two tasks (gate probability prediction and equivalent gate identification), reporting lower MAE as training data size increases. The central claims are that ForgeEDA exposes performance gaps among EDA tools and improves AI4EDA performance and generalization through scale and diversity.
Significance. If the dataset is released as described and the evaluation issues are fixed, ForgeEDA would be a valuable contribution. Its multi-representation coverage across RTL, post-mapping netlists, placed designs, and AIGs is broader than OpenABC-D or CircuitNet, and the scale (83,155 sub-AIGs) is substantially larger than existing public corpora. The resource could support benchmarking and pre-training for circuit representation learning. The use of standard EDA tools and the concrete generation pipeline are strengths. However, the significance is conditional on the evidence for generalization, which is currently not established because the training/evaluation protocol in Section V is underspecified.
major comments (3)
- [Section V.A.2, Tables IV and V] Tables IV and V are central to the claim that ForgeEDA improves generalization, but the evaluation protocol is not described. Section V.A.2 says only that 'we further divide our training dataset into 10% and 1% subsets'; it does not state whether the reported MAE values are computed on training or held-out data, nor the unit of the split. Because Section III.B.3 states that sub-AIGs are 'randomly extracted' from 4,450 parent AIGs, a split at the sub-AIG level can place sub-circuits of the same parent design in both training and test, leaking structural information and inflating scaling gains. The authors must specify a test split that is disjoint at the repository or AIG level, and must report held-out test MAE alongside training MAE.
- [Section IV.A, Table III] Table III's geomean and 'Imp.' rows are not internally consistent. The DCU geomean (48,523.05 µm², 14,604.15 ps) is computed over all 20 rows, whereas the Yosys geomean (71,987.59 µm², 15,944.40 ps) is computed over only the 16 benchmarks that Yosys completes, since four cells contain '–'. Moreover, the ratios of these geomeans are 71,987.59/48,523.05 ≈ 1.48 for area and 15,944.40/14,604.15 ≈ 1.09 for delay, not the reported 1.77× and 1.20×. The text should state exactly how 'Imp.' was computed, for example as a geomean of per-benchmark ratios over the 16 common benchmarks, and the numbers should be recalculated; as written, the performance-gap claim in Section IV.A is not reproducible.
- [Section III.B] The claim that ForgeEDA reflects 'real-world' chip design diversity is not yet supported by the reported construction. Section III.B.1 describes keyword-based repository collection and synthesizability filtering, but gives no provenance statistics, deduplication procedure, size distribution, or license documentation; Section III.B.2 then treats each module as a separate top module during synthesis. Consequently, modules from one repository become independent samples and the reported PPA statistics are module-level, not full-chip values. The paper should either provide quantitative evidence that the collection spans realistic complexity and diversity, such as gate counts, design hierarchy, and category balance, and explicitly discuss the module-as-top simplification as a limitation, or it should temper the 'comprehensive' and 'real-world' claims.
minor comments (7)
- [Section II.A] Section II.A says ForgeEDA covers '7 categories and 22 sub-categories of circuit designs,' but the Abstract, Introduction, Table I, and Conclusion all state 6 categories and 20 divisions; these counts should be reconciled.
- [Table I] Table I has a typo: 'Multipiler' should be 'Multiplier'; the affiliation list also spells 'Intergrated' for 'Integrated.'
- [Section IV.A] The text says 'we employed a variety of advanced open-source tools and commercial EDA solutions,' but the experiment compares only DCU and Yosys; the wording should match the actual scope.
- [Figures 2 and 3] Figures 2 and 3 do not define 'area reduction' and 'delay reduction' or report the per-benchmark values underlying the geometric-mean markers; definitions and raw data should be supplied.
- [Section V] The GNN experiments do not report hyperparameters, training epochs, random seeds, or model sizes, which are necessary for reproducing Tables IV and V.
- [Dataset Availability] No URL or release mechanism for the dataset is given; for an open-source dataset paper, a download link and usage instructions should be provided.
- [Tables IV and V] The column labels 'Loss Value' should say 'MAE' to match the equations and the text, which define the metric as mean absolute error.
Circularity Check
Data-scaling results in Tables IV/V are training-loss comparisons; the abstract's generalization claim is not independently tested.
-
fitted input called prediction
[Section V.A.2 (Experiment Setting) and V.A.3 (Result Analysis), Tables IV and V; also Eqs. (2) and (4)]
"We further divide our training dataset into 10% and 1% subsets to evaluate the data scalability of the models and assess the effectiveness of our expanded dataset. Table IV summarizes the performance results, where lower MAE indicates better performance."
The reported quantity in Tables IV/V is called 'loss value' and is computed by Eqs. (2)/(4) as MAE between the model prediction and the ground-truth label. The experiment section only describes subsets of 'our training dataset' and never defines a held-out test set, split unit, or evaluation split. On the present text, the table entries are therefore the training loss itself—the same objective minimized during training. Under that reading, the monotone decrease in loss from 1% to 10% to full dataset is expected by construction: a model trained on more of the scored examples fits them better, so the claimed 'as dataset size increases, MAE decreases ...
full rationale
The dataset construction and the practical EDA evaluations are externally grounded: RTL collection is from internet repositories, synthesis uses Synopsys DCU and Skywater 130nm PDK, placement uses Cadence Innovus, and AIG generation/STA uses ABC. The Yosys-vs-DCU and AIG-optimization comparisons do not reduce to the paper's own definitions or fitted parameters, so those parts are self-contained. The self-citations to DeepGate2, PolarGate, and DeepGate3 are used as baselines or background motivation; they are not the load-bearing evidence for the central claim once the AI4EDA evaluation is examined. The circularity is confined to the data-scaling demonstration in Tables IV/V: with no test split described, the reported MAE is the training objective, so the observed improvement with dataset size is a fitting artifact rather than evidence of generalization. This makes the abstract's 'improve model performance and generalization' claim partially circular, warranting a score of 6 rather than a higher score, because the dataset itself and the tool benchmarks retain independent content.
Assumptions & free parameters
assumptions (3)
- domain assumption The 1,189 Verilog repositories gathered from the internet using datasheet keywords and filtered for synthesizability are representative of real-world chip designs.
- domain assumption Treating each module as the top module in synthesis produces PPA metrics and netlists that are meaningful for benchmarking and training.
- domain assumption Randomly extracted sub-AIGs of 500 to 5,000 nodes are adequate training samples for circuit representation learning.
Cite this review
Pith. "Pith review of ForgeEDA: A Comprehensive Multimodal Dataset for Advancing EDA." pith.science (2026). https://pith.science/paper/PDM5LBUU
@misc{pith2026250502016,
author = {Pith},
title = {Pith review of: ForgeEDA: A Comprehensive Multimodal Dataset for Advancing EDA},
year = {2026},
howpublished = {\url{https://pith.science/paper/PDM5LBUU}},
note = {Machine review of arXiv:2505.02016}
}
read the original abstract
We introduce ForgeEDA, an open-source comprehensive circuit dataset across various categories. ForgeEDA includes diverse circuit representations such as Register Transfer Level (RTL) code, Post-mapping (PM) netlists, And-Inverter Graphs (AIGs), and placed netlists, enabling comprehensive analysis and development. We demonstrate ForgeEDA's utility by benchmarking state-of-the-art EDA algorithms on critical tasks such as Power, Performance, and Area (PPA) optimization, highlighting its ability to expose performance gaps and drive advancements. Additionally, ForgeEDA's scale and diversity facilitate the training of AI models for EDA tasks, demonstrating its potential to improve model performance and generalization. By addressing limitations in existing datasets, ForgeEDA aims to catalyze breakthroughs in modern IC design and support the next generation of innovations in EDA.
Figures
Forward citations
Cited by 3 Pith papers
-
Miter-Aware LUT Mapping: Aligning Structure and Solvability for Efficient Logic Equivalence Checking
Joint LUT mapping of golden and implementation circuits, combined with Gaussian-guided XOR modeling and solver-oriented LUT selection, reduces SAT-based logic equivalence checking runtime by up to 92.1%.
-
TRACE: Learning to Compute on Circuit Graphs
A per-operator hierarchical Transformer with a residual function shift objective substantially improves learning functional behavior of circuit graphs.
-
DeepCell: Self-Supervised Multiview Fusion for Circuit Representation Learning
DeepCell fuses AIG and post-mapping netlist views with masked autoencoding, achieving 2.77% lower ECO patch cost and 15-16% lower area-delay product in technology mapping.
Reference graph
Works this paper leans on
-
[1]
L.-T. Wang, Y .-W. Chang, and K.-T. T. Cheng,Electronic design automation: synthesis, verification, and test. Morgan Kaufmann, 2009
work page 2009
-
[2]
Machine learning for electronic design automation: A survey,
G. Huang, J. Hu, Y . He, J. Liuet al., “Machine learning for electronic design automation: A survey,”ACM Transactions on Design Automation of Electronic Systems (TODAES), vol. 26, no. 5, pp. 1–46, 2021
work page 2021
-
[3]
Large circuit models: opportunities and challenges,
L. Chen, Y . Chen, Z. Chuet al., “Large circuit models: opportunities and challenges,”Science China Information Sciences, vol. 67, no. 10, pp. 1–42, 2024
work page 2024
-
[4]
Notes on the iscas’89 bench- mark circuits,
F. Brglez, D. Bryan, and K. Kozminski, “Notes on the iscas’89 bench- mark circuits,”North-Carolina State University, 1989
work page 1989
-
[5]
Characteristics of the itc’99 benchmark circuits,
S. Davidson, “Characteristics of the itc’99 benchmark circuits,” inITSW, 1999
work page 1999
-
[6]
The epfl combinational benchmark suite,
L. Amar ´u, P.-E. Gaillardon, and G. De Micheli, “The epfl combinational benchmark suite,” inIWLS, no. CONF, 2015
work page 2015
-
[7]
Openabc-d: A large- scale dataset for machine learning guided integrated circuit synthesis,
A. B. Chowdhury, B. Tan, R. Karri, and S. Garg, “Openabc-d: A large- scale dataset for machine learning guided integrated circuit synthesis,” arXiv preprint arXiv:2110.11292, 2021
arXiv 2021
-
[8]
Z. Chai, Y . Zhao, Y . Lin, W. Liu, R. Wang, and R. Huang, “Circuitnet: An open-source dataset for machine learning applications in electronic design automation (eda),”arXiv preprint arXiv:2208.01040, 2022
arXiv 2022
Show all 46 references
-
[9]
Circuitnet 2.0: An advanced dataset for promoting machine learning innovations in realistic chip design environment,
X. Jiang, Y . Zhao, Y . Lin, R. Wang, R. Huanget al., “Circuitnet 2.0: An advanced dataset for promoting machine learning innovations in realistic chip design environment,” inICLR, 2024
2024
-
[10]
Benchmarking large language models for auto- mated verilog rtl code generation,
S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan- Gavitt, and S. Garg, “Benchmarking large language models for auto- mated verilog rtl code generation,” in2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2023, pp. 1–6
2023
-
[11]
Hardware model checking competition 2024,
A. Biere, N. Froleyks, and M. Preiner, “Hardware model checking competition 2024,” 2024
2024
-
[12]
Global Benchmark Database
M. Iser and C. Jabs, “Global Benchmark Database.” [Online]. Available: https://benchmark-database.de/
-
[13]
Hassoun and T
S. Hassoun and T. Sasao,Logic synthesis and verification. Springer Science & Business Media, 2001, vol. 654
2001
-
[14]
Scalable don’t-care-based logic optimization and resynthesis,
A. Mishchenko, R. Braytonet al., “Scalable don’t-care-based logic optimization and resynthesis,”ACM Transactions on Reconfigurable Technology and Systems (TRETS), vol. 4, no. 4, pp. 1–23, 2011
2011
-
[15]
Dag-aware aig rewriting a fresh look at combi- national logic synthesis,
A. Mishchenkoet al., “Dag-aware aig rewriting a fresh look at combi- national logic synthesis,” inDesign Automation Conference, 2006, pp. 532–535
2006
-
[16]
Delay-driven physically-aware logic synthesis with informed search,
L. Zhu and X. Guo, “Delay-driven physically-aware logic synthesis with informed search,” in2023 IEEE 41st International Conference on Computer Design (ICCD). IEEE, 2023, pp. 327–335
2023
-
[17]
Physically aware synthesis revisited: guiding technology mapping with primitive logic gate placement,
H. Pan, C. Lan, Y . Liu, Z. Wang, L. Shang, X. Zenget al., “Physically aware synthesis revisited: guiding technology mapping with primitive logic gate placement,”arXiv preprint arXiv:2408.07886, 2024
2024 arXiv
-
[18]
Masterrtl: A pre-synthesis ppa estimation framework for any rtl design,
W. Fang, Y . Lu, S. Liu, Q. Zhang, C. Xu, L. W. Wills, H. Zhang, and Z. Xie, “Masterrtl: A pre-synthesis ppa estimation framework for any rtl design,” in2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–9
2023
-
[19]
Sns’s not a synthesizer: a deep-learning-based synthesis predictor,
C. Xu, C. Kjellqvist, and L. W. Wills, “Sns’s not a synthesizer: a deep-learning-based synthesis predictor,” inInternational Symposium on Computer Architecture, 2022, pp. 847–859
2022
-
[20]
Retrieval-guided reinforcement learning for boolean circuit minimiza- tion,
A. B. Chowdhury, M. Romanelli, B. Tan, R. Karri, and S. Garg, “Retrieval-guided reinforcement learning for boolean circuit minimiza- tion,” inInternational Conference on Learning Representations
-
[21]
Circuitvae: Efficient and scalable latent circuit optimization,
J. Song, A. Swopeet al., “Circuitvae: Efficient and scalable latent circuit optimization,”arXiv preprint arXiv:2406.09535, 2024
2024 arXiv
-
[22]
Deeptpi: Test point insertion with deep reinforcement learning,
Z. Shi, M. Li, S. Khan, L. Wang, N. Wang, Y . Huang, and Q. Xu, “Deeptpi: Test point insertion with deep reinforcement learning,” in2022 IEEE International Test Conference (ITC). IEEE, 2022, pp. 194–203
2022
-
[23]
Efficient and reliable high-level synthesis design space explorer for fpgas,
D. Liu and B. C. Schafer, “Efficient and reliable high-level synthesis design space explorer for fpgas,” inInternational Conference on Field Programmable Logic and Applications. IEEE, 2016, pp. 1–8
2016
-
[24]
Accurate operation de- lay prediction for fpga hls using graph neural networks,
E. Ustun, C. Deng, D. Pal, Z. Li, and Z. Zhang, “Accurate operation de- lay prediction for fpga hls using graph neural networks,” ininternational conference on computer-aided design, 2020
2020
-
[25]
Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution,
S. Liu, W. Fang, Y . Luet al., “Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution,” in 2024 IEEE LLM Aided Design Workshop (LAD), 2024, pp. 1–5
2024
-
[26]
Deepgate: Learning neural representations of logic gates,
M. Li, S. Khan, Z. Shiet al., “Deepgate: Learning neural representations of logic gates,” inDesign Automation Conference, 2022, pp. 667–672
2022
-
[27]
Deepgate2: Functionality-aware circuit representation learning,
Z. Shi, H. Pan, S. Khan, M. Li, Y . Liu, J. Huanget al., “Deepgate2: Functionality-aware circuit representation learning,” in2023 IEEE/ACM International Conference on Computer Aided Design. IEEE, 2023
2023
-
[28]
Deepseq: Deep sequential circuit learning,
S. Khan, Z. Shi, M. Li, and Q. Xu, “Deepseq: Deep sequential circuit learning,” inDATE. IEEE, 2024, pp. 1–2
2024
-
[29]
Deepgate4: Efficient and effective representation learning for circuit design at scale,
Z. Zheng, S. Huang, J. Zhong, Z. Shi, G. Dai, N. Xu, and Q. Xu, “Deepgate4: Efficient and effective representation learning for circuit design at scale,”arXiv preprint arXiv:2502.01681, 2025
2025 arXiv
-
[30]
Deepgate3: Towards scalable circuit representation learning,
Z. Shi, Z. Zheng, S. Khan, J. Zhong, M. Li, and Q. Xu, “Deepgate3: Towards scalable circuit representation learning,” in2024 IEEE/ACM International Conference on Computer Aided Design. IEEE, 2024
2024
-
[31]
[Online]
Alldatasheet, 2003. [Online]. Available: https://www.alldatasheet.com/
2003
-
[32]
Deepcircuitx: A comprehensive repository-level dataset for rtl code understanding, generation, and ppa analysis,
Z. Li, C. Xu, Z. Shi, Z. Peng, Y . Liu, Y . Zhouet al., “Deepcircuitx: A comprehensive repository-level dataset for rtl code understanding, generation, and ppa analysis,”arXiv preprint arXiv:2502.18297, 2025
2025 arXiv
-
[33]
Deeprtl: Bridging verilog understanding and generation with a unified representation model,
Y . Liu, C. Xu, Y . Zhou, Z. Li, and Q. Xu, “Deeprtl: Bridging verilog understanding and generation with a unified representation model,”arXiv preprint arXiv:2502.15832, 2025
2025 arXiv
-
[34]
Introduction to the skywater pdk: The new age of open source silicon
T. Edwards, “Introduction to the skywater pdk: The new age of open source silicon.” [Online]. Available: https://skywater-pdk.readthedocs.io/
-
[35]
Abc: A system for sequential synthesis and verification
A. Mishchenkoet al., “Abc: A system for sequential synthesis and verification.”
-
[36]
Fast graph representation learning with pytorch geometric,
M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,”arXiv preprint arXiv:1903.02428, 2019
1903 arXiv
-
[37]
Design compiler ultra,
Synopsys, “Design compiler ultra,”https://www.synopsys.com/ implementation-and-signoff/rtl-synthesis-test/design-compiler-nxt.html, 2024
2024
-
[38]
Yosys open synthesis suite,
C. Wolf, “Yosys open synthesis suite,”https://github.com/YosysHQ/ yosys, 2016
2016
-
[39]
Delay optimiza- tion using sop balancing,
A. Mishchenko, R. Brayton, S. Jang, and V . Kravets, “Delay optimiza- tion using sop balancing,” inInternational Conference on Computer- Aided Design (ICCAD). IEEE, 2011, pp. 375–382
2011
-
[40]
Dag-aware synthesis orchestration,
Y . Liet al., “Dag-aware synthesis orchestration,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024
2024
-
[41]
Global delay optimization using structural choices,
A. Mishchenko, R. Brayton, and S. Jang, “Global delay optimization using structural choices,” inFPGA’10, 2010, pp. 181–184
2010
-
[42]
Polargate: Breaking the functionality representation bottleneck of and-inverter graph neural network,
J. Liu, J. Zhai, M. Zhao, Z. Lin, B. Yu, and C. Shi, “Polargate: Breaking the functionality representation bottleneck of and-inverter graph neural network,” inInternational Conference on Computer Aided Design, 2024
2024
-
[43]
Functionality matters in netlist representation learning,
Z. Wang, C. Bai, Z. He, G. Zhang, Q. Xu, T.-Y . Ho, B. Yu, and Y . Huang, “Functionality matters in netlist representation learning,” in Design Automation Conference, 2022, pp. 61–66
2022
-
[44]
Semi-supervised classification with graph convolutional networks,
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,”arXiv preprint arXiv:1609.02907, 2016
2016 arXiv
-
[45]
Graph attention networks,
P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y . Ben- gio, “Graph attention networks,”arXiv preprint arXiv:1710.10903, 2017
2017 arXiv
-
[46]
Inductive representation learning on large graphs,
W. Hamiltonet al., “Inductive representation learning on large graphs,” Advances in neural information processing systems, vol. 30, 2017
2017
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.