Pith. sign in

REVIEW 4 major objections 6 minor 16 references

ForgeHLS: A Large-Scale, Open-Source Dataset for High-Level Synthesis

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ForgeHLS contributes 459,850 high-level synthesis designs from 846 kernels, and models trained on it generalize better to unseen kernels than models trained on prior HLS datasets.

desk verdict A genuinely large, public HLS dataset that deserves peer review, but the synthetic half's validation is thinner than the headline number implies. read the letter →

arxiv 2507.03255 v3 pith:CDP7BVIV submitted 2025-07-04 cs.AR cs.AI

Zedong Peng , Zeju Li , Mingzhe Gao , Qiang Xu , Chen Zhang , Jieru Zhao This is my paper · ORCID
classification cs.ARcs.AI
keywords high-levelsynthesisHLSdatasetdesignspaceexplorationQoRpredictionpragmainsertionmachinelearningFPGALLMfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ForgeHLS, a dataset of 459,850 HLS design points generated from 846 C++ kernels, roughly ten times larger than prior public HLS datasets. The dataset is built by combining curated real-world kernels with LLM-generated synthetic kernels, then automatically inserting pragma configurations (loop unrolling, pipelining, array partitioning) through exhaustive and Bayesian design-space exploration. The paper's central claim is that this scale and diversity lets machine-learning models trained on ForgeHLS predict post-synthesis quality (latency and resource counts) on kernels never seen during training better than the same models trained on earlier datasets, and lets fine-tuned LLMs insert pragmas with higher synthesis success rates and closer to Pareto-optimal configurations.

What carries the argument

The carrier of the argument is the dataset itself, produced by an automated pragma-insertion workflow. A static analysis pass extracts loop trip counts, array sizes, and the top function from each kernel; a tree-based explorer enumerates pragma combinations under HLS validity rules (e.g., no inner unroll inside a pipelined outer loop, factors restricted to powers of two), and a Bayesian optimizer guides exploration for kernels whose design space is too large to enumerate. Each retained design bundles the original code, the inserted pragmas, the post-synthesis resource and latency report, and a control-dataflow graph, which is what makes the dataset usable for both GNN- and LLM-based learning.

What would settle it

Take a random sample of about 200 synthetic kernels from S-ForgeHLS, run their C code with concrete inputs alongside an RTL-level co-simulation of the same kernels, and compare outputs. If a non-negligible fraction (say, above 5%) diverge, or if removing the divergent kernels materially changes the QoR-prediction generalization results reported in the paper, the central claim that the dataset's synthetic portion is reliable would be weakened.

Watch

Extended reading notes

Core claim

ForgeHLS is the claim that a sufficiently large and diverse set of synthesized HLS designs can lift machine-learning-based HLS optimization from overfitting to generalization. The paper reports that GNNs and fine-tuned LLMs trained on ForgeHLS achieve substantially lower prediction error on a held-out set of unseen kernels than models trained on previous datasets, and that fine-tuned LLMs raise the fraction of valid pragma insertions (synthesis pass rate) and reduce the average distance to the Pareto-optimal latency-resource frontier. The paper presents ForgeHLS as a resource that allows the community to train models for two standard downstream tasks: QoR prediction and automated pragma exploration.

Load-bearing premise

The load-bearing premise is that the roughly 360,000 synthetic designs in S-ForgeHLS are functionally correct and representative, since they are validated by an LLM-generated Python reference implementation and a single HLS syntax pass rather than full RTL-level functional equivalence checks.

Editorial extensions

If this is right

  • If the claim holds, ML-based QoR prediction can be trained on an order-of-magnitude larger and more diverse design space, moving from kernel-specific fitting to cross-kernel generalization.
  • Fine-tuned LLMs become a viable route for automated pragma insertion, raising the HLS synthesis pass rate for generated designs.
  • The dataset provides a common benchmark with kernel-level splits, enabling fair comparison of future QoR and design-space-exploration methods.
  • The inclusion of post-HLS artifacts such as control-dataflow graphs, reports, and Verilog allows GNN-based and text-based models to be compared on identical design points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The one-step pragma-exploration protocol in the paper likely underestimates what LLMs could achieve in multi-step, iterative design-space exploration; the dataset's size and structure make such protocols directly testable.
  • Because the synthetic kernels are generated from a curated algorithm list, the dataset's diversity is bounded by that list; expanding the list through further generation is a testable way to extend coverage.
  • The functional validation of synthetic designs depends on the LLM-as-oracle; as LLM code generation improves, a similar pipeline could produce larger verified subsets, but the paper's current numbers should be re-checked as the oracle changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ForgeHLS, a large-scale HLS dataset containing 459,850 designs derived from 846 C++ kernels. The dataset is split into R-ForgeHLS (99,601 designs from established benchmarks such as CHStone, MachSuite, PolyBench, Rosetta, and Vitis collections) and S-ForgeHLS (360,249 designs from GPT-4o-generated synthetic kernels). The authors describe a Clang-based automated pragma insertion workflow with two exploration strategies, a 'full DSE' tree-based explorer and a Bayesian DSE, and release post-HLS artifacts including reports, CDFGs, and Verilog. They evaluate the dataset on two downstream tasks: GNN/LLM-based QoR prediction on standard and kernel-unseen splits, and LLM-based automatic pragma insertion measured by HLS pass rate and ADRS relative to dataset-defined Pareto fronts. The central claim is that ForgeHLS improves scale, diversity, and design coverage over prior HLS datasets, and that models trained on it generalize better to unseen kernels.

Significance. If the dataset is as valid and diverse as claimed, it would be a substantial community resource: it is roughly an order of magnitude larger than existing HLS datasets in design count, it ships open-source code and public data, it includes graph-level and report-level artifacts useful for GNN and LLM research, and it documents over 200k CPU hours of construction effort. The paper honestly reports some limitations, notably Appendix G's admission that the 'full DSE' is not exhaustive, and the evaluation code is reproducible in principle. The main scientific value is the resource itself; the utility demonstrations are useful but currently more suggestive than conclusive because of validation and statistical issues described below.

major comments (4)
  1. [§3.3 vs Appendix G] Section 3.3 states that the automated workflow explores 'the full design space by generating all possible combinations of pragma types and factors,' yet Appendix G explicitly states that 'the full DSE does not exhaustively enumerate all pragma permutations' because it restricts factors to powers of two, disallows inner-loop unrolling under an outer pipeline, and prunes configurations with equivalent factors. These statements describe different design spaces. Please reconcile the terminology, define the actual enumerated design space precisely, and quantify how much of the unrestricted space is covered after pruning. Without this, the 'full DSE' and coverage claims in the abstract and Section 3.1 are ambiguous and potentially overstated.
  2. [Appendix F / Table 2] The functional validation of S-ForgeHLS (360,249 designs, 78% of the dataset) compares the GPT-4o-generated C++ against a Python implementation also produced by GPT-4o from the same algorithm description. Since both artifacts derive from the same prompt and the same model, agreement does not establish that the C++ matches the intended algorithm; the 'oracle' is not independent. In addition, the single-pass Vitis HLS run checks syntactic/synthesis acceptability, not RTL-level semantic equivalence. Please provide either (a) an independent functional check on a statistically meaningful sample of synthetic kernels—for example, cross-compiling the C++ with a non-LLM reference implementation or golden test vectors—and report the error rate with confidence intervals, or (b) a clear statement that S-ForgeHLS is only syntax-validated and adjust the 'valid HLS designs' claim accordingly. The current evidence is too weak to support the headline count of 459,850 functionally valid designs.
  3. [§4.1, Unseen Test] The 'Unseen Test Set' is constructed by holding out 10% of ForgeHLS kernels, so the claimed generalization to 'unseen kernels' is in-distribution with respect to the ForgeHLS generation pipeline; it does not demonstrate transfer to kernels outside that distribution. The comparison in Tables 3 and 4 is therefore not a test of generalization to genuinely external kernels. Please add an external held-out evaluation (e.g., a benchmark suite not used in training or a random set of independently written HLS kernels), or soften the generalization claims in the abstract and Section 4.1 accordingly.
  4. [§4.2, ADRS and Tables 3–5] The utility demonstrations currently lack error bars: Tables 3–5 report single-split results with no repeated seeds, no confidence intervals, and no significance tests, although differences between configurations are often small (e.g., MAPE values around 0.04–0.05 in the unseen columns of Table 4). Additionally, the ADRS metric in Section 4.2 measures distance to the 'exact Pareto-optimal set of kernel k in ForgeHLS,' but Appendix G establishes that the explored design space is not exhaustive for large kernels, so the reference Pareto set is only the Pareto front of the sampled designs, not the true design space. Please report variability across runs and clarify that ADRS is relative to the dataset-defined Pareto front.
minor comments (6)
  1. [Appendix A / Eq. (4)] ARU is defined twice, once in Eq. (3) in Section 4.2 and again in Eq. (4) in Appendix A; please unify the definitions to avoid confusion.
  2. [Appendix D, Listing 1] The JSON example contains typos: 'Avialable_BRAM_18K' should be 'Available_BRAM_18K', and there is a missing comma after the 'latency-resource-strategy' value before the 'source_code' key.
  3. [Table 4] The Mistral-7B-v0.2 rows for HLSyn and HLSDataset on the Standard Test are numerically identical; please check whether this is a copy/paste error.
  4. [§4.2] The 'high/medium/low resource usage' classification is described as the top/middle/bottom third by ARU, but the exact cutoff points are not given; please provide the thresholds or the code used to compute them.
  5. [Appendix H] The CLI example uses a flag '--bayesian_opt_number 25' while Algorithm 2 describes Nopt and Ncalls; please align the naming between the pseudocode and the released tool.
  6. [Table 2 caption] The caption says 'Avg. #P' is the average number of pragmas per kernel, but for R-ForgeHLS this value is 192.4 while 'Avg. #T' is 1298.2; please clarify the units in the caption so readers do not mistake per-kernel counts for per-design counts.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-referential validation in the synthetic-data path (GPT-4o acts as both code generator and functional oracle), but the dataset construction and downstream benchmarks are externally grounded in Vitis HLS and independent baselines.

  1. other [Section 3.2 and Appendix F (Validate Synthetic Data)]
    "We prompt GPT-4o with the algorithm specifications and HLS design rules, which generates the corresponding kernel code in C, devoid of pragmas. ... To validate the functional accuracy, we employ a cross-verification approach leveraging GPT-4o as an assistant. Specifically, we provided GPT-4o with an algorithm description and the corresponding implementation, requesting it to generate a Python version of the same algorithm. This Python implementation served as a reference model for validation."

    The functional-correctness gate for S-ForgeHLS (360,249 of the 459,850 designs per Table 2) compares GPT-4o-generated C++ kernels against a Python reference that is also generated by GPT-4o from the same algorithm description. The reference is therefore not independent: any systematic misunderstanding of the description by GPT-4o is shared by both implementations, so passing the check establishes self-consistency rather than conformance to the intended algorithm. This is a validation circularity, not a derivational one, but it directly undergirds the headline 'valid HLS designs' count for 78% of the dataset.

full rationale

The central dataset is not derived from the downstream experiments: ForgeHLS is built by inserting pragmas into kernels and running Vitis HLS, an external synthesis tool, to obtain post-HLS reports, graphs, and validity. The QoR evaluation compares models trained on ForgeHLS with models trained on external datasets (DB4HLS, HLSyn, HLSDataset, HLSFactory) and with a prior GNN baseline, providing independent benchmark support. No fitted parameter is renamed as a prediction, and the self-citations (Gao et al. 2024; Li et al. 2025) supply methodology or a kernel-specification list rather than a load-bearing uniqueness theorem. The only notable self-referentiality is in the synthetic-data validation: GPT-4o generates both the C++ kernel and the Python oracle used to check it, so systematic model errors are invisible to the check. In addition, the Task 2 Pareto 'ground truth' is defined as the Pareto front within ForgeHLS's own generated designs, and the 'Unseen Test' is an internal kernel-level split of ForgeHLS, so those evaluations measure in-distribution consistency of the dataset rather than externally certified optimality. These are genuine limitations of external validity but they do not reduce the paper's derivation to its own inputs; the score is therefore low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

None of the central claims depend on fitted scientific constants, and the paper introduces no new physical entities. The free parameters listed are human-chosen construction and evaluation choices that shape the dataset's coverage and the Pareto ground truth. The load-bearing assumptions are about toolchain reliability, the adequacy of the pragma pruning rules, and the validity of using GPT-4o as both kernel generator and functional reference.

free parameters (2)
  • Bayesian DSE search budget = 20 random starts + 40 iterations (also referenced as --bayesian_opt_number 25)
    Chosen in Appendix H; controls which configurations are retained for kernels too large for full enumeration, and therefore shapes the coverage statistics that support the diversity claim.
  • ARU resource weights = equal weights over BRAM, FF, LUT, DSP
    Eq. 3 and Eq. 4 define the Pareto designs used in Task 2; equal weighting is a hand-set choice and changes the Pareto frontier if altered.
assumptions (4)
  • domain assumption Vitis HLS reports (worst/best latency, BRAM, DSP, FF, LUT) are accurate and deterministic labels for each design.
    Dataset labels and all downstream metrics come from these reports; no RTL simulation or repeated-run variance is reported (Appendix I, Section 4).
  • domain assumption Automated pragma insertion rules preserve valid designs and do not exclude important configurations.
    Appendix G restricts factors to powers of two and prunes branches, e.g. no inner unroll when outer loop is pipelined; no ablation shows the pruned regions are unimportant.
  • ad hoc to paper GPT-4o can serve as a functional correctness oracle for synthetic kernels.
    Appendix F uses GPT-4o-generated Python implementations and random tests to accept or reject GPT-4o-generated C++ code; this assumes the Python reference is correct.
  • domain assumption A kernel-level 10% split is a fair measure of generalization to unseen kernels.
    Section 4.1 relies on non-overlapping kernel names; it does not analyze source-level similarity across benchmark families or splits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ForgeHLS: A Large-Scale, Open-Source Dataset for High-Level Synthesis." pith.science (2026). https://pith.science/paper/CDP7BVIV

@misc{pith2026250703255,
  author       = {Pith},
  title        = {Pith review of: ForgeHLS: A Large-Scale, Open-Source Dataset for High-Level Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDP7BVIV}},
  note         = {Machine review of arXiv:2507.03255}
}
read the original abstract

High-Level Synthesis (HLS) plays a crucial role in modern hardware design by transforming high-level code into optimized hardware implementations. However, progress in applying machine learning (ML) to HLS optimization has been hindered by a shortage of sufficiently large and diverse datasets. To bridge this gap, we introduce ForgeHLS, a large-scale, open-source dataset explicitly designed for ML-driven HLS research. ForgeHLS comprises over 400k diverse designs generated from 846 kernels covering a broad range of application domains, consuming over 200k CPU hours during dataset construction. Each kernel includes systematically automated pragma insertions (loop unrolling, pipelining, array partitioning), combined with extensive design space exploration using Bayesian optimization. Compared to existing datasets, ForgeHLS significantly enhances scale, diversity, and design coverage. We further define and evaluate representative downstream tasks in Quality of Result (QoR) prediction and automated pragma exploration, clearly demonstrating ForgeHLS utility for developing and improving ML-based HLS optimization methodologies. The dataset and code are public at https://github.com/zedong-peng/ForgeHLS.

Figures

Figures reproduced from arXiv: 2507.03255 by the authors.

Figure 1
Figure 1. The overview workflow of our dataset construction and evaluation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Box plot distribution comparison of post-HLS feature across HLS datasets. Metrics include LUT (Look-Up Table), [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Breakdown of code in ForgeHLS, weighted by [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Design space coverage across existing datasets. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Design space coverage of ForgeHLS across different data sources [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Detailed distribution of ForgeHLS across data source. Each dot represents one individual design. Each box shows [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The structure of generator and filter [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The process of full design space exploration: (a) showcases an input algorithm written in C++ with a nested loop [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

  1. [8]

    Automatic Hardware Pragma Insertion in High-Level Synthesis: A Non-Linear Programming Approach

    Auto- matic Hardware Pragma Insertion in High-Level Synthe- sis: A Non-Linear Programming Approach. arXiv preprint arXiv:2405.12304. First article. Pouget, S.; Pouchet, L.-N.; and J.Cong

    work page Pith review arXiv

  2. [9]

    A Unified Framework for Automated Code Transformation and Pragma Insertion

    Enhanc- ing High-Level Synthesis with Automated Pragma Inser- tion and Code Transformation Framework. arXiv preprint arXiv:2405.03058. Second article. Reagen, B.; Adolf, R.; Shao, Y . S.; Wei, G.-Y .; and Brooks, D

    work page Pith review arXiv

  3. [12]

    AutoDSE: Enabling Software Programmers to Design Efficient FPGA Accelerators

    AutoDSE: Enabling Software Programmers to Design Effi- cient FPGA Accelerators. arXiv:2009.14381. Sohrabizadeh, A.; et al

    work page Pith review arXiv 2009

  4. [13]

    In 2023 IEEE 34th International Con- ference on ASAP, 197–204

    HLSDataset: Open-source dataset for ML-assisted FPGA design using high level synthesis. In 2023 IEEE 34th International Con- ference on ASAP, 197–204. IEEE. Wu, N.; Xie, Y .; and Hao, C

    work page 2023

  5. [14]

    In Proceedings of the 2021 on Great Lakes Symposium on VLSI, 39–44

    Ironman: GNN-assisted Design Space Exploration in High-Level Synthesis via Re- inforcement Learning. In Proceedings of the 2021 on Great Lakes Symposium on VLSI, 39–44. IEEE. Wu, N.; et al

    work page 2021

  6. [16]

    This includes loop trip counts, loop locations, array sizes and dimensions, array definition locations, and the identification and location of the top function

    The process begins with Clang, which performs a syntactic analysis of the C++ source code to extract essential information needed for DSE. This includes loop trip counts, loop locations, array sizes and dimensions, array definition locations, and the identification and location of the top function. Such analysis ensures that the appropriate pragmas can be...

    work page 2023

  7. [2008]

    In 2008 IEEE International Symposium on Circuits and Systems (ISCAS) , 1192–1195

    Chstone: A benchmark program suite for practical c-based high-level synthesis. In 2008 IEEE International Symposium on Circuits and Systems (ISCAS) , 1192–1195. IEEE. Hu, E. J.; Shen, Y .; Wallis, P.; Allen-Zhu, Z.; Li, Y .; Wang, S.; Wang, L.; and Chen, W

    work page 2008

  8. [2009]

    In 2009 International Symposium on VLSI Design, Automation and Test, 106–109

    Adaptive simulated annealer for high level synthesis design space exploration. In 2009 International Symposium on VLSI Design, Automation and Test, 106–109. IEEE. Sohrabizadeh, A.; Yu, C. H.; Gao, M.; and Cong, J

    work page 2009

Show all 16 references
  1. [2014]

    In 2014 IEEE IISWC , 110–119

    MachSuite: Benchmarks for accelerator design and customized architectures. In 2014 IEEE IISWC , 110–119. IEEE. Schafer, B. C.; Takenaka, T.; and Wakabayashi, K

    2014

  2. [2016]

    https://sourceforge.net/projects/polybench/files/polybench- c-4.2/

    PolyBench/C 4.2. https://sourceforge.net/projects/polybench/files/polybench- c-4.2/. Accessed: 2025-07-29. Pouget, S.; Pouchet, L.-N.; and Cong, J

    2025

  3. [2018]

    In Proceed- ings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, 269–278

    Rosetta: A realistic high-level synthesis bench- mark suite for software programmable FPGAs. In Proceed- ings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, 269–278. A Design Space Coverage across Datasets. . As shown in Fig. 4, our dataset de...

    2018

  4. [2020]

    In 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), 574–580

    HL-Pow: A Learning-Based Power Modeling Framework for High- Level Synthesis. In 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), 574–580. Pouchet, L.-N.; and Yuki, T

    2020

  5. [2021]

    arXiv:2106.09685

    LoRA: Low-Rank Adap- tation of Large Language Models. arXiv:2106.09685. Kuang, H.; Cao, X.; Li, J.; and Wang, L

    arXiv

  6. [2023]

    In 2023 ICFPT, 106–114

    HGBO- DSE: Hierarchical GNN and Bayesian Optimization based HLS Design Space Exploration. In 2023 ICFPT, 106–114. Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y .; Zheng, L.; Yu, C. H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I

    2023

  7. [2024]

    In Pro- ceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, 1–9

    HLSFactory: A Framework Empowering High-Level Syn- thesis Datasets for Machine Learning and Beyond. In Pro- ceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, 1–9. Bai, Y .; Sohrabizadeh, A.; Qin, Z.; Hu, Z.; Sun, Y .; and Cong, J

    2024

  8. [2025]

    arXiv:2502.18297

    DeepCircuitX: A Comprehensive Repository- Level Dataset for RTL Code Understanding, Generation, and PPA Analysis. arXiv:2502.18297. Lin, Z.; Yuan, Z.; Zhao, J.; Zhang, W.; Wang, H.; and Tian, Y

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.