{"id":"e5d53638-c756-425a-9dca-105db5b3cc72","arxiv_id":"2505.02016","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ForgeEDA introduces a large multimodal circuit dataset spanning RTL code, post-mapping netlists, placed netlists, and AIGs, with small benchmark experiments on synthesis tools and AI4EDA models.","lead":"ForgeEDA is a new open-source dataset of 1,189 Verilog chip designs with synthesized netlists, placed netlists, and logic graphs, meant to train AI models and evaluate EDA tools. The paper shows benchmark runs on synthesis and graph learning, but the dataset download link is not provided and several evaluation details are missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tables IV/V may report training-set loss rather than held-out generalization: no test split is described, and with sub-AIGs randomly extracted from parent AIGs, the data-scaling gains could reflect memorization or leakage instead of diversity.","rationale":"The paper is best read as a resource contribution: 1,189 RTL repositories with multiple downstream representations and 83,155 sub-AIGs is a useful offer to the EDA community regardless of the AI experiments. I take the strongest claim to be the abstract's statement that ForgeEDA's scale and diversity improve AI model performance and generalization. That claim is supported only by Tables IV/V, and the manuscript never describes a held-out evaluation: Section V.A.2 says the authors divide the training dataset into 10% and 1% subsets, and the tables report 'loss value' without specifying train/test status or split unit. If the loss is training loss, the monotonically decreasing MAE is expected from more capacity or memorization and says nothing about generalization. If there is a held-out split, the absence of repository-disjoint grouping remains a risk because sub-AIGs come from parent AIGs that come from modules of the same repositories. This does not make the dataset invalid; it makes the central AI4EDA demonstration unverified as written. I agree with the reader only partially: the reader's weakest assumption concerned source representativeness and transfer to full-chip workloads, while I see the evaluation protocol as the more immediate gate. The condition for acceptance should be to release split metadata and recompute the two tables with a repository-disjoint held-out test set. The final verdict remains conditional, hence UNCHANGED.","tokens_in":9628,"tokens_out":7819,"duration_ms":85342,"concrete_test":"Obtain the ForgeEDA split-generation code and dataset metadata from the authors, or ask them to state the exact evaluation protocol for Tables IV and V. Then re-run both experiments with a strict held-out test set: all sub-AIGs from entire source repositories (or at minimum entire AIGs) held out from training, with the model evaluated only on that test set and MAE reported over multiple seeds. If the full-versus-1% MAE gap persists under this protocol, the scaling claim stands; if it collapses, the reported gains are due to training-set loss and/or leakage rather than dataset diversity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ForgeEDA improves AI model performance and generalization rests on Tables IV and V (Section V). In both tasks, the paper says only that 'we further divide our training dataset into 10% and 1% subsets' (Section V.A.2); it never specifies a held-out test set, the split unit, or how the reported 'loss value' is computed. If the loss is computed on the training data, then lower error on the full dataset is an expected memorization effect, not evidence of generalization. Even if a test set exists, the 83,155 sub-AIGs are 'randomly extracted' from 4,450 AIGs (Section III.B.3) and modules are synthesized independently (Section III.B.2), so without repository-disjoint or AIG-disjoint splitting, subgraphs of the same parent design can appear in both training and test, inflating scaling gains. Since the abstract's strongest claim is about generalization, the absence of any documented evaluation protocol is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ForgeEDA is a proposed open-source multimodal circuit dataset. It collects 1,189 Verilog repositories from the internet based on datasheet-derived keywords, filters them for synthesizability, and synthesizes each module as a top module with Synopsys Design Compiler and the SkyWater 130nm PDK, producing 4,450 post-mapping netlists, 4,450 placed netlists, 4,450 AIGs, and 83,155 sub-AIGs with PPA and timing reports. The paper benchmarks DCU against Yosys on RTL synthesis, compares several ABC synthesis and mapping scripts, and trains five GNN models on two tasks (gate probability prediction and equivalent gate identification), reporting lower MAE as training data size increases. The central claims are that ForgeEDA exposes performance gaps among EDA tools and improves AI4EDA performance and generalization through scale and diversity.","tokens_in":9830,"tokens_out":13217,"duration_ms":111194,"significance":"If the dataset is released as described and the evaluation issues are fixed, ForgeEDA would be a valuable contribution. Its multi-representation coverage across RTL, post-mapping netlists, placed designs, and AIGs is broader than OpenABC-D or CircuitNet, and the scale (83,155 sub-AIGs) is substantially larger than existing public corpora. The resource could support benchmarking and pre-training for circuit representation learning. The use of standard EDA tools and the concrete generation pipeline are strengths. However, the significance is conditional on the evidence for generalization, which is currently not established because the training/evaluation protocol in Section V is underspecified.","major_comments":[{"comment":"Tables IV and V are central to the claim that ForgeEDA improves generalization, but the evaluation protocol is not described. Section V.A.2 says only that 'we further divide our training dataset into 10% and 1% subsets'; it does not state whether the reported MAE values are computed on training or held-out data, nor the unit of the split. Because Section III.B.3 states that sub-AIGs are 'randomly extracted' from 4,450 parent AIGs, a split at the sub-AIG level can place sub-circuits of the same parent design in both training and test, leaking structural information and inflating scaling gains. The authors must specify a test split that is disjoint at the repository or AIG level, and must report held-out test MAE alongside training MAE.","section":"Section V.A.2, Tables IV and V"},{"comment":"Table III's geomean and 'Imp.' rows are not internally consistent. The DCU geomean (48,523.05 µm², 14,604.15 ps) is computed over all 20 rows, whereas the Yosys geomean (71,987.59 µm², 15,944.40 ps) is computed over only the 16 benchmarks that Yosys completes, since four cells contain '–'. Moreover, the ratios of these geomeans are 71,987.59/48,523.05 ≈ 1.48 for area and 15,944.40/14,604.15 ≈ 1.09 for delay, not the reported 1.77× and 1.20×. The text should state exactly how 'Imp.' was computed, for example as a geomean of per-benchmark ratios over the 16 common benchmarks, and the numbers should be recalculated; as written, the performance-gap claim in Section IV.A is not reproducible.","section":"Section IV.A, Table III"},{"comment":"The claim that ForgeEDA reflects 'real-world' chip design diversity is not yet supported by the reported construction. Section III.B.1 describes keyword-based repository collection and synthesizability filtering, but gives no provenance statistics, deduplication procedure, size distribution, or license documentation; Section III.B.2 then treats each module as a separate top module during synthesis. Consequently, modules from one repository become independent samples and the reported PPA statistics are module-level, not full-chip values. The paper should either provide quantitative evidence that the collection spans realistic complexity and diversity, such as gate counts, design hierarchy, and category balance, and explicitly discuss the module-as-top simplification as a limitation, or it should temper the 'comprehensive' and 'real-world' claims.","section":"Section III.B"}],"minor_comments":[{"comment":"Section II.A says ForgeEDA covers '7 categories and 22 sub-categories of circuit designs,' but the Abstract, Introduction, Table I, and Conclusion all state 6 categories and 20 divisions; these counts should be reconciled.","section":"Section II.A"},{"comment":"Table I has a typo: 'Multipiler' should be 'Multiplier'; the affiliation list also spells 'Intergrated' for 'Integrated.'","section":"Table I"},{"comment":"The text says 'we employed a variety of advanced open-source tools and commercial EDA solutions,' but the experiment compares only DCU and Yosys; the wording should match the actual scope.","section":"Section IV.A"},{"comment":"Figures 2 and 3 do not define 'area reduction' and 'delay reduction' or report the per-benchmark values underlying the geometric-mean markers; definitions and raw data should be supplied.","section":"Figures 2 and 3"},{"comment":"The GNN experiments do not report hyperparameters, training epochs, random seeds, or model sizes, which are necessary for reproducing Tables IV and V.","section":"Section V"},{"comment":"No URL or release mechanism for the dataset is given; for an open-source dataset paper, a download link and usage instructions should be provided.","section":"Dataset Availability"},{"comment":"The column labels 'Loss Value' should say 'MAE' to match the equations and the text, which define the metric as mean absolute error.","section":"Tables IV and V"}],"recommendation":"major_revision","confidential_remarks":"The dataset has clear potential value, and the two main technical concerns are fixable: the missing held-out test protocol in Section V and the inconsistent geomean/ratio calculations in Table III. I would ask the authors to provide the dataset link and the internal-consistency fixes before acceptance. The category-count discrepancy and the missing release instructions are also worth checking in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nForgeEDA is a real contribution to the EDA open-data landscape. The paper delivers a multimodal bundle — 1,189 Verilog repositories, 4,450 post-mapping netlists, 4,450 placed netlists, 4,450 AIGs, and 83,155 sub-AIGs — synthesized with commercial tools on a real PDK. That is a larger and more varied testbed than OpenABC-D or CircuitNet, and it gives the community a common ground for benchmarking synthesis and training circuit-learning models. The construction pipeline is concrete: datasheet-derived keywords, synthesizability filtering, per-module top synthesis, ABC for AIGs. I believe the dataset itself is valuable and will be used.\n\nThe soft spots are real but concentrated. The biggest one is the AI4EDA evaluation (Section V). The paper says “we further divide our training dataset into 10% and 1% subsets” — no test split is described anywhere. The reported numbers are called “loss value”, and with no held-out set, the scaling gains in Tables IV and V could be memorization rather than generalization. The sub-AIGs are randomly extracted from parent AIGs, so without a design-disjoint or repository-disjoint split, the same parent can leak into both training and evaluation. This directly undercuts the abstract's “generalization” claim. The stress-test note is right.\n\nSecond, Table III compares Yosys on 16 benchmarks against DCU on 20. The geomean ratio 1.77x area / 1.20x delay is over different benchmark sets; the fair comparison would be on the 16 common designs. It's fixable, and probably the qualitative conclusion survives, but as written the numbers are not apples-to-apples.\n\nThird, I could not find a download link or repository URL in the paper. For a dataset paper that is a major omission — the artifact is the contribution.\n\nMinor: the category count is inconsistent (6 categories / 20 divisions in the intro, 7 categories / 22 sub-categories in Related Work).\n\nWho is this for? Anyone building or benchmarking AI models for logic synthesis, PPA prediction, or circuit representation learning. The dataset is the contribution, and it deserves a serious referee. The paper should be conditionally accepted only after the authors document the evaluation protocol, provide the artifact link, and fix the geomean comparison. I'd want to see held-out numbers before believing the scaling claims.\n\nMy recommendation: send it out for peer review, but ask the authors to ship the data and rerun the AI experiments with a proper split. It is worth the referee time.","headline":"A useful multimodal EDA dataset whose AI scaling claims are undercut by the lack of a documented held-out evaluation split.","tokens_in":10467,"tokens_out":2762,"would_cite":true,"duration_ms":25329,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new open dataset of 1,189 chip designs provides 83,155 logic graphs and shows more training data improves AI for EDA.","keywords":["ForgeEDA","open-source dataset","logic synthesis","AI for EDA","circuit representation learning","power performance area","And-Inverter Graphs","data scaling"],"falsifier":"A concrete check is to retrain a circuit model on a matched set of full-chip synthesized netlists: if the data-scaling gains seen on ForgeEDA's module-level sub-AIGs disappear, the module-as-top-level shortcut is not a valid proxy for real design diversity.","tokens_in":9443,"feed_emoji":"⚙️","tokens_out":7619,"duration_ms":74858,"temperature":0.7,"pith_summary":"The paper introduces ForgeEDA, a large open-source multimodal dataset for electronic design automation (EDA). It collects 1,189 Verilog repository designs spanning processors, AI accelerators, arithmetic units, encoders/decoders, interfaces, and controllers, then runs synthesis and physical design flows to produce 4,450 post-mapping netlists, 4,450 placed netlists, and 4,450 And-Inverter Graphs, plus 83,155 sub-graphs for model training. The authors argue that existing open EDA datasets are too small or too narrow, and that their dataset's scale and diversity let researchers benchmark synthesis tools and train circuit-learning models more fairly. On benchmarks, open-source synthesis tools fall behind commercial tools by about 1.77x in area and 1.20x in delay on average, and graph-neural-network models for circuit learning improve steadily as training data grows from 1% to the full dataset. The paper's central claim is that this public infrastructure will close the data gap that has slowed AI for EDA.","feed_headline":"Chip-design AI gets 83,155 new logic graphs in open dataset","feed_subtitle":"ForgeEDA pairs RTL code with netlists, placed layouts, and logic graphs to train and stress-test EDA models.","key_machinery":"The central object is the dataset itself, with its four linked circuit representations: RTL source code, post-mapping netlists, placed netlists, and And-Inverter Graphs, along with 83,155 randomly extracted sub-AIGs of 500-5,000 nodes stored as graph representations for PyTorch Geometric. The mechanism that makes it useful is the explicit alignment of all formats and their accompanying synthesis, placement, and timing reports to a single design, so the same circuit can be studied at each stage of the EDA flow. This alignment supports both benchmarking of EDA tools on identical input/output pairs and supervised training of circuit-learning models on graph structure plus PPA labels.","core_discovery":"ForgeEDA is a multimodal circuit dataset that binds four representations of each design into one resource: RTL source code, post-mapping netlists, placed netlists with physical-design reports, and And-Inverter Graphs. The construction expands coverage by collecting repositories via datasheet-derived keywords and by synthesizing each module as a separate top module, turning 1,189 repositories into 4,450 netlists. The paper evaluates the dataset in two ways: it benchmarks logic synthesis and AIG optimization, exposing a substantial performance gap between open-source and commercial tools, and it trains representative circuit-learning models on the graphs, showing that model error decreases monotonically as the training subset grows from 1% to 10% to the full dataset. The conclusion is that the scale and diversity of training data are decisive for AI4EDA and that ForgeEDA supplies the missing resource.","pith_inferences":["Because each module is synthesized separately as a top module, the netlists are mostly block-level rather than full-system; gains from training on sub-AIGs may not transfer to whole-chip netlists, and a full-chip companion set would be a natural extension.","The monotone data-scaling curves invite a formal study of scaling laws for circuit representation learning, potentially predicting how much data a target error requires.","All PPA reports come from one 130nm process design kit, so models trained on them may need fine-tuning to transfer to other process nodes or FPGA mappings.","The paired RTL code and PPA labels could support fine-tuning large language models to generate synthesizable RTL that is aware of downstream area and delay, an application the paper does not explore."],"forward_implications":["Researchers can use ForgeEDA's 4,450 netlists and 83,155 sub-AIGs to train and evaluate circuit representation models, with evidence that larger training subsets lower prediction error.","Logic-synthesis benchmarking is no longer confined to near-saturated small benchmarks; the dataset reveals that open-source synthesis tools lag commercial tools by roughly 1.77x in area and 1.20x in delay on average.","The multimodal alignment lets users study how design decisions propagate across synthesis and physical design stages, including timing reports extracted after placement.","Public release gives researchers a common ground for comparing PPA prediction and logic-optimization methods, potentially exposing where academic tools fail on real-world designs."],"supporting_citations":[{"why":"Provides OpenABC-D, the main existing multi-format dataset whose limited number of designs ForgeEDA claims to surpass.","marker":"[7]"},{"why":"Offers CircuitNet's multi-stage physical-design data but only for processors, motivating the need for broader coverage.","marker":"[8]"},{"why":"Extends CircuitNet to CircuitNet 2.0, still constrained to processor-focused designs, reinforcing the coverage gap.","marker":"[9]"},{"why":"Demonstrates that circuit representation models improve with more data, motivating ForgeEDA's data-scaling experiments.","marker":"[30]"},{"why":"Supplies the datasheet-derived keywords used to collect the 1,189 RTL repositories across diverse categories.","marker":"[31]"},{"why":"Provides the ABC tool that converts PM netlists into AIGs and runs AIG optimization and timing analysis.","marker":"[35]"},{"why":"Supplies the PyTorch Geometric graph data structures used to represent sub-AIGs for model training.","marker":"[36]"}],"fun_headline_variants":["ForgeEDA pairs 4 circuit views to train and stress-test EDA AI","Open dataset links RTL, netlists, and graphs for EDA models","4-in-1 circuit dataset exposes open vs commercial EDA gap","ForgeEDA: 4450 netlists from 1189 repos for AI4EDA","Multimodal EDA dataset: RTL, netlists, AIGs, placed layouts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's comprehensiveness rests on the assumption that the 1,189 synthesizable Verilog repositories collected from the internet via datasheet keywords are representative of real-world chip design diversity, while treating each module as a separate top module assumes module-level synthesis statistics are meaningful proxies for full-chip PPA.","fun_headline_variants_meta":{"raw":{"variants":["ForgeEDA pairs 4 circuit views to train and stress-test EDA AI","Open dataset links RTL, netlists, and graphs for EDA models","4-in-1 circuit dataset exposes open vs commercial EDA gap","ForgeEDA: 4450 netlists from 1189 repos for AI4EDA","Multimodal EDA dataset: RTL, netlists, AIGs, placed layouts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2679,"prompt_tokens":862,"completion_tokens":1817,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1710}},"tokens_in":478,"tokens_out":1817,"duration_ms":10997,"temperature":1.0,"reasoning_tokens":1710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:04:21.755689+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to retrain a circuit model on a matched set of full-chip synthesized netlists: if the data-scaling gains seen on ForgeEDA's module-level sub-AIGs disappear, the module-as-top-level shortcut is not a valid proxy for real design diversity.","supporting_citations":[{"cited_title":"Circuitnet 2.0: An advanced dataset for promoting machine learning innovations in realistic chip design environment,","cited_arxiv_id":null,"evidence_quote":"Extends CircuitNet to CircuitNet 2.0, still constrained to processor-focused designs, reinforcing the coverage gap."},{"cited_title":"Deepgate3: Towards scalable circuit representation learning,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that circuit representation models improve with more data, motivating ForgeEDA's data-scaling experiments."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Supplies the datasheet-derived keywords used to collect the 1,189 RTL repositories across diverse categories."},{"cited_title":"Abc: A system for sequential synthesis and verification","cited_arxiv_id":null,"evidence_quote":"Provides the ABC tool that converts PM netlists into AIGs and runs AIG optimization and timing analysis."}],"review_version":1}